Automated phenotyping of behavior

Abstract

The systems and methods described herein provide techniques for automating phenotyping. In some embodiments, these systems and methods process video data, identify body parts, extract frame-level features, and determine subject behavior captured in the video data. These systems and methods can use one or more machine learning models.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 058,569, filed July 30, 2020, pursuant to 35 U.S. SC §119(e), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] In some aspects, the present invention relates to automated phenotypic analysis of behavior, particularly noxious behavior, by processing video data.

[0004] Government support

[0005] This invention was developed with government support under projects R21DA048634 and UM1OD023222 funded by the National Institutes of Health (NIH). The U.S. government owns certain rights to this invention. Background Technology

[0006] Pain is a complex combination of bodily information, emotional context, and personal subjective experience. Animal subjects are often used to study the neural and genetic basis of pain behavior. Because the subjective experience of animals cannot be measured, other methods have been developed to quantify "nociceptive" behaviors, defined as behavioral responses to painful stimuli. Many nociceptive assays rely on rapid withdrawal reflexes in response to brief mechanical or thermal stimuli; these simple movements are relatively easy to define and identify, but they lack similarity to clinical pain. In mice, these assays correlate poorly with more clinically relevant chronic pain assays, but are more closely associated with startle and reactive traits. Formalin assays have previously been developed to assess nociceptive behavior in rats and to monitor complex behaviors in response to chemically induced localized inflammation over long periods. Currently available formalin assays are limited in effectiveness due to their reliance on human observers to identify when animals exhibit nociceptive behavior. This reliance makes observation labor-intensive, time-consuming, and highly subjective, as different nociceptive behaviors are not always uniformly defined and recorded. Summary of the Invention

[0007] According to one aspect of the present invention, a computer-implemented method is provided, the method comprising: receiving video data representing video-captured motion of a subject; determining first point data for a first frame during a first time period using the video data, the first point data identifying the location of a first body part of the subject; determining second point data for the first frame using the video data, the second point data identifying the location of a second body part of the subject; determining first distance data using the first point data and the second point data, the first distance data representing the distance between the first body part and the second body part; determining a first feature vector corresponding at least to the first frame and the second frame, the first feature vector including at least the first distance data and the second distance data; processing at least the first feature vector using a trained model configured to identify the probability that the subject will exhibit a behavior during the first time period; and determining a first label corresponding to the first time period based on the processing of at least the first feature vector, the first label identifying a first behavior of the subject during the first time period. In some embodiments, the method further includes determining third point data for the first frame using the video data, the third point data identifying the location of a third body part of the subject. In some embodiments, the method further includes determining second distance data using first point data and third point data, the second distance data representing the distance between a first body part and a third body part; determining a second feature vector corresponding to a first frame that includes at least the second distance data; and wherein processing using a trained model includes processing the first feature vector and the second feature vector using the trained model. In some embodiments, the method further includes determining first angle data using the first point data, second point data, and third point data, the first angle data representing the angle corresponding to the first body part, second body part, and third body part; determining a second feature vector corresponding at least to a first frame, the second feature vector including at least the first angle data; and wherein processing using a trained model further includes processing the first feature vector and the second feature vector using the trained model.In some embodiments, the method further includes determining fourth point data for a second frame during a first time period using video data, the fourth point data identifying the position of a first body part; determining fifth point data for the second frame using the video data, the fifth point data identifying the position of a second body part; determining sixth point data for the second frame using the video data, the sixth point data identifying the position of a third body part; determining third distance data for the second frame using the fourth and fifth point data, the third distance data representing the distance between the first and second body parts; determining fourth distance data for the second frame using the fourth and sixth point data, the fourth distance data representing the distance between the first and third body parts; determining second angle data for the second frame using the fourth, fifth, and sixth point data, the second angle data representing the angle corresponding to the first, second, and third body parts; and determining a second feature vector including at least the third distance data, the fourth distance data, and the second angle data. In some embodiments, the second distance data represents the distance between the first and second body parts for the second frame during the first time period. In some embodiments, the method further includes using at least first distance data and second distance data to compute metric data corresponding to a first frame, wherein a first feature vector includes the metric data. In some embodiments, the metric data represents a statistical analysis corresponding at least to the first distance data and the second distance data, the statistical analysis being at least one of mean, standard deviation, median, and median absolute deviation. In some embodiments, the method further includes processing video data using an additional trained model to determine first point data, wherein the first point data includes pixel data representing the location of a first body part. In some embodiments, the method further includes processing video data using an additional trained model to determine the probability that pixel coordinates correspond to a first body part; and determining the first point data, including pixel coordinates, based at least in part on the probability of satisfying a threshold. In some embodiments, the method further includes using video data to determine additional point data for the first frame, the additional point data identifying the locations of at least 12 parts of the subject, wherein these 12 parts include at least a first body part and a second body part. In some embodiments, the method further includes determining additional distance data for the first frame, the additional distance data representing distances between multiple pairs of body parts formed using paired portions of the subject's 12 parts, wherein the first feature vector includes the additional distance data. In some embodiments, the method further includes determining additional angle data for the first frame, the additional angle data representing angles corresponding to multiple body part triples formed by selecting three portions from the subject's 12 parts, wherein the first feature vector includes the additional angle data.In some embodiments, the method further includes: determining additional feature vectors corresponding to six frames during a first time period, the six frames including at least the first and second frames; using these additional feature vectors to compute metric data representing at least one of the mean, standard deviation, median, and median absolute deviation; and processing the metric data using a trained model to determine a first label. In some embodiments, the method further includes determining location data for the first frame, representing pixel coordinates of 12 parts of the subject, the location data including at least first point data, second point data, and third point data, and wherein processing the metric data using a trained model further includes processing the location data using the trained model. In some embodiments, the method further includes determining additional feature vectors corresponding to 11 frames during the first time period, the 11 frames including at least the first and second frames; using these additional feature vectors to compute metric data representing at least one of the mean, standard deviation, median, and median absolute deviation; and processing the metric data using a trained model to determine a first label. In some embodiments, the method further includes the following 11 frames: five frames before the first frame and five frames after the first frame. In some embodiments, the method further includes determining additional feature vectors corresponding to 21 frames during a first time period, the 21 frames including at least the first and second frames; using these additional feature vectors to compute metric data representing at least one of the mean, standard deviation, median, and median absolute deviation; and processing the metric data using a trained model to determine a first label. In some embodiments, the 21 frames include 11 frames preceding the first frame and 11 frames following the first frame. In some embodiments, the video data represents video capture motion of more than one subject. In some embodiments, receiving video data includes receiving video data obtained from a top-down video of the subject. In some embodiments, receiving video data includes receiving video data not obtained from a top-down video of the subject. In some embodiments, the video includes top-down video. In some embodiments, the video includes video different from top-down video. In some embodiments, the video does not include top-down video. In some implementations, the trained model is a classifier configured to process feature data corresponding to video frames to determine the behavior exhibited by a subject as presented in those video frames, the feature data corresponding to multiple parts of the subject. In some implementations, the first body part is the subject's mouth; the second body part is the subject's right hind foot; the trained model is configured to identify the likelihood that the subject exhibits contact between the first and second body parts; and a first label indicates that the first frame represents contact between the first and second body parts. In some implementations, the first frame corresponds to 30 milliseconds of video data.In some embodiments, the video data corresponds to a first video capturing a top-view view of the subject and a second video capturing a side view of the subject. In some embodiments, the subject is a mammal. In some embodiments, the subject is a rodent. In some embodiments, the subject is a primate. In some embodiments, the subject is a genetically engineered subject. In some embodiments, the subject is a mouse. In some embodiments, the mouse is a genetically engineered rodent. In some embodiments, the subject is a genetically engineered mouse.

[0008] According to another aspect of the invention, a method for determining the nociceptive behavior of a test subject is provided, the method comprising monitoring the test subject's response, wherein the monitoring means comprises a computer-implemented method as described in any embodiment of any of the foregoing methods. In some embodiments, the test subject exhibits a pain condition. In some embodiments, the pain condition includes one or more of the following: inflammatory pain, neuropathic pain, myalgia, arthritis, chronic pain, visceral pain, cancer pain, and postoperative pain. In some embodiments, the test subject is an animal model exhibiting a pain condition. In some embodiments, pain is induced in the test subject. In some embodiments, inducing pain includes exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, and chemical agents. In some embodiments, the means of inducing pain in the test subject includes inducing inflammation in the test subject. In some embodiments, the means of inducing inflammation includes exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, and chemical agents. In some embodiments, the chemical agents include one or more of formalin and acetone. In some embodiments, the means of inducing pain include one or more of the following: exposing the test subject to a noxious stimulus (e.g., a pain-inducing chemical agent), implanting an element that generates a noxious stimulus into the subject, injecting the test subject with a pain-inducing chemical agent, etc. In some embodiments, the test subject is a genetically engineered test subject. In some embodiments, the test subject is a rodent, optionally a mouse. In some embodiments, the test subject is a genetically engineered rodent. In some embodiments, the test subject is a genetically engineered mouse. In some embodiments, the method further includes administering a candidate therapeutic agent to the test subject. In some embodiments, if pain is induced in the test subject, the candidate therapeutic agent is administered to the test subject prior to the induction of pain. In some embodiments, if pain is induced in the test subject, the candidate therapeutic agent is administered to the test subject during and after the induction of pain. In some embodiments, the candidate therapeutic agent is administered to the test subject once. In some embodiments, the candidate therapeutic agent is administered to the test subject two, three, four, five, or more times. In some embodiments, the candidate therapeutic agent is administered to the test subject two or more times, all before, all during, or all after pain induction. In some embodiments, the candidate therapeutic agent is administered to the test subject two or more times in a combination of two or more of the following: administration before, administration during, and administration after pain induction. In some embodiments, the monitoring results of the test subjects are compared with control results. In some embodiments, control results are from control subjects monitored using computer-implemented methods.In some embodiments, pain is induced in control subjects. In some embodiments, the pain induced in control subjects is substantially equivalent to the pain induced in test subjects. In some embodiments, control subjects are animal models exhibiting pain syndrome. In some embodiments, both test animals and control animals are animal models exhibiting pain syndrome. In some embodiments, the test animal model and control animal model are the same animal model. In some embodiments, the test animal model and control animal model are different animal models. In some embodiments, control subjects are not administered the candidate therapeutic agent. In some embodiments, the amount (also referred to as the "dosage") of the candidate therapeutic agent administered to the control subject differs from the amount (dosage) of the candidate therapeutic agent administered to the test subject. In some embodiments, the control results are the result of prior monitoring of the test subject using a computer-implemented method. In some embodiments, the monitoring of the subject identifies the subject's chronic pain syndrome. In some embodiments, the monitoring of the subject identifies the efficacy of the candidate therapeutic agent in treating the pain syndrome.

[0009] According to another aspect of the invention, a method is provided for identifying the efficacy of a candidate therapeutic agent in treating a pain condition in a subject. The method includes: administering the candidate therapeutic agent to a test subject; and monitoring the test subject, wherein the monitoring means include a computer-implemented method of any embodiment of any of the foregoing aspects of the invention; and wherein monitoring results indicating pain relief in the test subject identify the efficacy of the candidate therapeutic agent in treating the pain condition. In some embodiments, the pain condition includes one or more of the following: inflammatory pain, neuropathic pain, myalgia, arthritis, chronic pain, visceral pain, cancer pain, and postoperative pain. In some embodiments, the test subject has a pain condition. In some embodiments, the test subject is an animal model of a pain condition. In some embodiments, pain is induced in the test subject prior to monitoring. In some embodiments, inducing pain in the test subject includes inducing inflammation in the test subject. In some embodiments, inducing inflammation includes exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, and chemical agents. In some embodiments, the chemical agent includes one or more of formalin and acetone, and the means of inducing pain include one or more of the following: exposing the test subject to the chemical agent, inserting the chemical agent into the test subject, implanting the chemical agent into the test subject, and injecting the chemical agent into the test subject. In some embodiments, the means of inducing pain in the subject include one or more of the following: exposing the subject to an effective amount of the pain-inducing agent. In some embodiments, the means of inducing pain in the subject include one or more of the following: exposing the subject to an effective amount of the pain-inducing chemical agent; implanting the pain-inducing agent (such as, but not limited to, stimulating electrodes, sustained-release chemicals, etc.) into or onto the subject; injecting the pain-inducing chemical agent into the subject, etc. In some embodiments, the test subject is a genetically engineered test subject. In some embodiments, the test subject is a rodent, optionally a mouse. In some embodiments, the test subject is a genetically engineered rodent. In some embodiments, the test subject is a genetically engineered mouse. In some embodiments, the candidate therapeutic agent is administered to the test subject prior to inducing pain in the test subject. In some embodiments, the candidate therapeutic agent is administered to the test subject during and after pain induction. In some embodiments, the candidate therapeutic agent is administered to the test subject once. In some embodiments, the candidate therapeutic agent is administered to the test subject two, three, four, five, or more times. In some embodiments, the two or more administrations of the candidate therapeutic agent to the test subject are all performed before, all during, or all after pain induction.In some embodiments, the administration of the candidate therapeutic agent to the test subject two or more times is a combination of two or more of the following: administration before pain induction, administration during pain induction, and administration after pain induction. In some embodiments, monitoring results of the test subject are compared with control results. In some embodiments, control results are from control subjects monitored using computer-implemented methods. In some embodiments, pain is induced in the control subject. In some embodiments, the pain induced in the control subject is substantially equivalent to the pain induced in the test subject. In some embodiments, the control subject has a pain condition and optionally is an animal model of a pain condition. In some embodiments, both the test animal and the control animal are animal models of a pain condition. In some embodiments, the control subject is not administered the candidate therapeutic agent. In some embodiments, the test animal model and the control animal model are the same animal model. In some embodiments, the test animal model and the control animal model are different animal models. In some embodiments, the amount (also referred to as a “dosage”) of the candidate therapeutic agent administered to the control subject differs from the amount or dose of the candidate therapeutic agent administered to the corresponding test subject. In some implementations, the control results are the results of prior monitoring of test subjects using computer-based methods. In some implementations, the method also includes additional testing of the efficacy of the candidate therapeutic agent.

[0010] According to another aspect of the invention, a system is provided, comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive video data representing video-captured motion of a subject; determine first point data for a first frame during a first time period using the video data, the first point data identifying the position of a first body part of the subject; determine second point data for the first frame using the video data, the second point data identifying the position of a second body part of the subject; determine first distance data using the first point data and the second point data, the first distance data representing the distance between the first body part and the second body part; determine a first feature vector corresponding at least to the first frame and the second frame, the first feature vector including at least the first distance data and the second distance data; process at least the first feature vector using a trained model configured to identify the probability that the subject will exhibit a behavior during the first time period; and determine a first label corresponding to the first time period based on the processing of at least the first feature vector, the first label identifying a first behavior of the subject during the first time period. In some embodiments, the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: determine third point data for the first frame using the video data, the third point data identifying the position of a third body part of the subject. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine second distance data using first point data and third point data, the second distance data representing the distance between a first body part and a third body part; determine a second feature vector corresponding to a first frame that includes at least the second distance data; and wherein the instructions causing the system to process using a trained model further cause the system to process the first feature vector and the second feature vector using the trained model. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine first angle data using first point data, second point data, and third point data, the first angle data representing the angle corresponding to the first body part, the second body part, and the third body part; determine a second feature vector corresponding at least to a first frame, the second feature vector including at least the first angle data; and wherein the instructions causing the system to process using a trained model further cause the system to process the first feature vector and the second feature vector using the trained model.In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine fourth point data using video data for a second frame during a first time period, the fourth point data identifying the position of a first body part; determine fifth point data using the video data for the second frame, the fifth point data identifying the position of a second body part; determine sixth point data using the video data for the second frame, the sixth point data identifying the position of a third body part; determine third distance data using the fourth and fifth point data for the second frame, the third distance data representing the distance between the first and second body parts; determine fourth distance data using the fourth and sixth point data for the second frame, the fourth distance data representing the distance between the first and third body parts; determine second angle data using the fourth, fifth, and sixth point data for the second frame, the second angle data representing the angle corresponding to the first, second, and third body parts; and determine a second feature vector including at least the third distance data, the fourth distance data, and the second angle data. In some embodiments, the second distance data represents the distance between the first and second body parts for the second frame during the first time period. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: compute metric data corresponding to a first frame using at least first distance data and second distance data, wherein a first feature vector includes the metric data. In some embodiments, the metric data represents a statistical analysis corresponding at least to the first distance data and second distance data, the statistical analysis being at least one of mean, standard deviation, median, and median absolute deviation. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: process video data using an additional trained model to determine first point data, wherein the first point data includes pixel data representing the location of a first body part. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: process video data using an additional trained model to determine the probability that pixel coordinates correspond to a first body part; and determine the first point data, including pixel coordinates, at least in part based on the probability of satisfying a threshold. In some embodiments, at least one memory also includes instructions that, when executed by at least one processor, further cause the system to: determine additional point data for a first frame using video data, the additional point data identifying the locations of at least 12 parts of the subject, wherein the 12 parts include at least a first body part and a second body part.In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine additional distance data for a first frame, the additional distance data representing distances between multiple pairs of body parts formed using pairs of parts from the subject's 12 parts, wherein a first feature vector includes the additional distance data. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine additional angle data for the first frame, the additional angle data representing angles corresponding to multiple body part triplets formed by selecting three parts from the subject's 12 parts, wherein a first feature vector includes the additional angle data. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine additional feature vectors corresponding to six frames during a first time period, the six frames including at least the first frame and the second frame; use these additional feature vectors to compute metric data representing at least one of mean, standard deviation, median, and median absolute deviation; and process the metric data using a trained model to determine a first label. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine location data for a first frame, representing pixel coordinates of 12 portions of a subject, the location data including at least first point data, second point data, and third point data, wherein the instructions causing the system to process the metric data using a trained model further cause the system to process the location data using a trained model. In some embodiments, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine additional feature vectors corresponding to 11 frames during a first time period, the 11 frames including at least the first frame and the second frame; use these additional feature vectors to compute metric data representing at least one of the mean, standard deviation, median, and median absolute deviation; and use the trained model to process the metric data to determine a first label. In some embodiments, the 11 frames include five frames preceding the first frame and five frames following the first frame. In some implementations, at least one memory further includes instructions that, when executed by at least one processor, further cause the system to: determine additional feature vectors corresponding to 21 frames during a first time period, the 21 frames including at least the first frame and the second frame; use these additional feature vectors to compute metric data representing at least one of the mean, standard deviation, median, and median absolute deviation; and process the metric data using a trained model to determine a first label.In some embodiments, the 21 frames include 11 frames before the first frame and 11 frames after the first frame. In some embodiments, the video data represents video capture motion of more than one subject. In some embodiments, the trained model is a classifier configured to process feature data corresponding to the video frames to determine the behavior exhibited by the subject presented in these video frames, the feature data corresponding to multiple parts of the subject. In some embodiments, the first body part is the subject's mouth; the second body part is the subject's right hind foot; the trained model is configured to identify the probability that the subject exhibits contact between the first and second body parts; and a first label indicates that the first frame represents contact between the first and second body parts. In some embodiments, the first frame corresponds to 30 milliseconds of video data. In some embodiments, the video data corresponds to a first video capturing a top view of the subject and a second video capturing a side view of the subject. In some embodiments, the video data includes video data obtained from a top-down video of the subject. In some embodiments, the video data includes video data as a side-view video of the subject. In some embodiments, the subject is a mammal. In some embodiments, the subject is a rodent. In some embodiments, the subject is a primate. In some embodiments, the subject is a genetically engineered subject. In some embodiments, the subject is a rodent, optionally a mouse. In some embodiments, the subject is a genetically engineered rodent. In some embodiments, the subject is a genetically engineered mouse.

[0011] According to another aspect of the invention, one or more non-transitory computer-readable media are provided, comprising computer-executable instructions that, when executed, cause at least one processor to perform actions including: receiving video data representing video-captured motion of a subject; determining first point data for a first frame during a first time period using the video data, the first point data identifying the position of a first body part of the subject; determining second point data for the first frame using the video data, the second point data identifying the position of a second body part of the subject; determining first distance data using the first point data and the second point data, the first distance data representing the distance between the first body part and the second body part; determining a first feature vector corresponding at least to the first frame and the second frame, the first feature vector including at least the first distance data and the second distance data; processing at least the first feature vector using a trained model configured to identify the probability that the subject exhibits a behavior during the first time period; and determining a first label corresponding to the first time period based on the processing of at least the first feature vector, the first label identifying a first behavior of the subject during the first time period. In some embodiments, receiving video data includes receiving video data obtained from one or both of a top-down video of the subject and a side-view video of the subject. In some embodiments, receiving video data includes receiving video data that is not obtained from a top-down video of the subject. In some embodiments, the subject is a genetically engineered test subject. In some embodiments, the subject is a rodent, optionally a mouse. In some embodiments, the subject is a genetically engineered rodent. In some embodiments, the subject is a genetically engineered mouse. Attached Figure Description

[0012] For a more complete understanding of this disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0013] Figure 1 This is a conceptual diagram of a system for determining subject behavior according to an embodiment of this disclosure.

[0014] Figure 2 This is a flowchart illustrating a process for analyzing video data of subjects to determine subject behavior according to an embodiment of this disclosure.

[0015] Figure 3 Example images of videos used by the system to determine the behavior of the subjects are shown.

[0016] Figure 4A This is a conceptual diagram of a test mouse in which various body parts are marked for system detection according to an embodiment of this disclosure.

[0017] Figure 4BIt is a conceptual diagram showing various body parts of a test mouse as described in the embodiments of this disclosure, as detected by the system.

[0018] Figure 5 and Figure 6 This is a flowchart illustrating a process for generating feature vectors according to an embodiment of this disclosure.

[0019] Figure 7A This is a conceptual diagram illustrating distance data according to an embodiment of the present disclosure, which represents the distance between two body parts of a subject during multiple video frames.

[0020] Figure 7B This is a conceptual diagram illustrating angle data according to an embodiment of the present disclosure, which represents the angles between three body parts of a subject during multiple video frames.

[0021] Figure 8 This is a flowchart illustrating the process for classifying subject behavior according to an embodiment of this disclosure.

[0022] Figure 9 Annotated images of videos showing the behavior of subjects analyzed by the system are presented, including system annotations and human annotations identifying various body parts of the subjects.

[0023] Figure 10 A table is shown according to an embodiment of this disclosure, illustrating the contribution of various window sizes to different body parts used by the system to analyze video data.

[0024] Figure 11A and Figure 11B A data graph showing the system for determining subject behavior per second in two short videos and the human annotators identifying subject behavior is presented. Figure 11A and Figure 11B In the model, the results for "licking" and "no licking" are indicated by crosshairs, while the human results are represented by solid lines. Figure 11B In the text, among the two pairs of vertically arranged + symbols, the upper + is used for model results, and the lower + is used for human results. NL stands for "No Licking".

[0025] Figure 12 The figure shows a data plot showing the percentage of consistency between the subject behaviors determined by the system and the subject behaviors identified by human annotators for multiple videos of four subjects in four activity locations.

[0026] Figure 13The diagram shows a comparison of labels manually assigned by two human annotators to three test mice with labels determined by the system for the same three mice. Dots represent system-determined labels; squares represent labels manually assigned by human annotator "Observer 1"; and triangles represent labels manually assigned by human annotator "Observer 2".

[0027] Figure 14 and 15A to Figure 15D This diagram shows data comparing the licking behavior of male and female mice from different strains. Figure 14 In the graph, the solid bars on the left represent female C57BL / 6J mice; the small square bars on the left represent female C57BL / 6NJ mice; the solid bars on the right represent male C57BL / 6J mice; and the large square bars on the right represent male C57BL / 6NJ mice. Figure 15A In the diagram, squares represent female C57BL / 6J mice, and triangles represent female C57BL / 6NJ mice. Figure 15B In the diagram, squares represent male C57BL / 6J mice, and inverted triangles represent male C57BL / 6NJ mice. Figure 15C and Figure 15D In the diagram, squares represent females and dots represent males.

[0028] Figure 16 This is a block diagram conceptually illustrating example components of a device according to an embodiment of this disclosure.

[0029] Figure 17 This is a block diagram conceptually illustrating example components of a server according to an embodiment of this disclosure.

[0030] Figure 18A The conceptual diagram shown illustrates the development and testing of a comprehensive injury sensitivity score according to an embodiment of this disclosure. Figures 18B to 18H The data graph shown illustrates the development and testing of the comprehensive nociceptiveness score. Figures 18B to 18F The data plot shown illustrates a measure of consistency between the classifier and human ratings (labelers). Figures 18G to 18H The data plots shown illustrate the effects of dose-based and strain-based methods on male mice ( Figure 18G ) and female mice ( Figure 18H The comparison is based on the time taken for licking; the dot represents strain C57BL6J, the triangle represents strain AJ, the "X" represents strain BALBcJ, and the asterisk represents strain C3HHeJ.

[0031] Figures 19A to 19GThe data plots and graphs shown present measurements of the time taken for paw tremors, uprighting, and stasis episodes, with comparisons made between male and female mice from strains C57BL6 / J, AJ, BALBcJ, and C3HHeJ. Figure 19A and Figure 19B Male mice were shown. Figure 19A ) and female mice ( Figure 19B The time taken for the claws to tremble due to tension. Figure 19C and Figure 19D Male mice were shown. Figure 19C ) and female mice ( Figure 19D The time spent standing upright due to tension. Figure 19E The table shows the accuracy measurement results. Figure 19F and Figure 19G Male mice were shown. Figure 19F ) and female mice ( Figure 19G The time taken for the attack to freeze, lasting 3 to 6 seconds. Figures 19A to 19D and Figures 19F to 19G In each of these, a dot represents strain C57BL6J, a triangle represents strain AJ, an "X" represents strain BALBcJ, and an asterisk represents strain C3HHeJ.

[0032] Figure 20 The heatmaps shown illustrate strain-specific differences in all measures, confirming the correlation between each measure and formalin dosage across the four mouse strains. Measures were categorized into four types: (1) behavioral classifier measures from the JAX Animal Behavior System (JABS), [Kabra, M. et al., January 2013, Nature Methods; 10(1):64-7, the contents of which are incorporated herein by reference in their entirety]; (2) engineered heuristic measures; (3) gait attitude measures derived from deep neural networks; and (4) classical open-field measures derived from tracking and classification. The intensity of the color corresponds to the magnitude of the correlation coefficient between the measure and dosage, either positive (^) or negative (*). Only statistically significant correlations are shown.

[0033] Figures 21A to 21D The presented data graphs demonstrate the construction of the univariate pain scale and the cross-validation evaluation of the binary logistic regression model. Figure 21A The line graph shown illustrates the use of Figure 22 The feature subsets in the data are fitted to a cumulative link model (logit link) of the sequential response (dose). Solid dots indicate the line plotting dose level 0; gray solid lines indicate the line plotting dose level 1; solid squares indicate the line plotting dose level 2; and asterisks indicate the line plotting dose level 3. Figure 21BThe bar chart shown illustrates the contribution of each feature to the univariate pain scale. For the importance of each feature's contribution, ** indicates highly important, * indicates important, and · indicates implied contribution. Figure 21C The data plot shown illustrates an animal leave-one-out cross-validation evaluation of the accuracy metrics of binary logistic regression models constructed using different feature sets, including: “open field” (gray solid dots); “other”, which includes engineered features and features obtained from the behavior classifier (“X”); and “all”, which includes the open field and both of the other two (“*”). Figure 21D The data plot shown illustrates the strain leave-one-out cross-validation evaluation of the accuracy metrics of binary logistic regression models constructed using different feature sets, including: “open field” (gray solid dots); “other”, which includes engineered features and features obtained from the behavior classifier (“X”); and “all”, which includes the open field and both of the other two (“*”). Figure 21C and Figure 21D The error bars in the table represent the bootstrap confidence intervals of the parameters.

[0034] Figure 22 Information is provided regarding video features of various behaviors observed in certain embodiments of the method of the present invention. Detailed Implementation

[0035] This invention partially includes a method utilizing video recording and machine learning techniques, comprising three parts: keypoint detection, per-frame feature extraction using these keypoints, and algorithmic classification of the behavior. This automated method is flexible because it can use different model classifiers or keypoints with only a small loss in accuracy. The method and system of this invention include a machine learning scoring system and provide the required accuracy, consistency, and ease of use, allowing the use of noxious stimulus-based systems (a non-limiting example being formalin determination), a viable option for large-scale genetic studies. The method and system of this invention provide a reliable and scalable automated scoring system for noxious behavior, significantly reducing the time and labor costs associated with noxious experiments and also reducing variability in such experiments.

[0036] Some aspects of the methods and systems of the present invention include the automated measurement of licking actions in a widely used open field activity setting with a top-down camera view. Some embodiments of the methods and systems of the present invention include the automated measurement of multiple possible anti-harm behaviors for the purpose of obtaining a comprehensive nociceptibility index. The methods of the present invention can be used to effectively quantify multiple anti-harm behaviors in video. In some embodiments of the methods and systems of the present invention, top-down video of each mouse during a one-hour open field activity is collected according to a previously disclosed protocol. Figure 1A). [See Kumar, V. et al., PNAS 108, 15557-15564, (2011), and Geuther, B. et al., Communications Biology 2, 124 (March 2019), all of which are incorporated herein by reference in their entirety]. Open field videos were processed using pose estimation and tracking networks based on deep neural networks to generate a 12-point pose skeleton and elliptical fitted trajectory for mice per frame [Sheppard, K. et al., bioRxiv 424780 (2020), and Geuther, B. et al., Communications Biology 2, 124 (March 2019), all of which are incorporated herein by reference in their entirety]. Using JABS, these per-frame measurements were used to construct a behavior classifier. These per-frame measurements were also used to engineer features such as traditional open field measurements of anxiety and hyperactivity, as well as neural network-based grooming and novel gait measurements. All features are being added. Figure 22 The methods and systems of this invention can also be used to assess genetic variations in nociceptive responses. As a non-limiting example, male and female mice were selected from strains ranging from known high-level licking responders (C57BL6J and C3HHeJ) to low-level licking responders (BALBcJ and AJ), and were examined using the methods and systems of this invention, with the results then compared.

[0037] Pain is a complex combination of bodily information, emotional context, and personal subjective experience. Animal subjects are often used to study the neural and genetic basis of pain behavior. Because the subjective experience of animals cannot be measured, other methods have been developed to quantify “nociceptive” behaviors (defined as behavioral responses to painful stimuli) and “injury prevention” behaviors (defined as behaviors associated with avoiding injury and harm). Many nociceptive assays rely on rapid withdrawal reflexes in response to brief mechanical or thermal stimuli; these simple movements are relatively easy to define and identify, but they lack similarity to clinical pain. In mice, these assays correlate poorly with more clinically relevant chronic pain assays, but are more strongly associated with startle and reactive traits. In contrast, the formalin test, originally developed for rats, is designed to monitor complex behaviors in response to chemically induced localized inflammation over extended periods. Typically, irritating formalin is injected into a hind paw, and the animal’s nociceptive behaviors, such as licking, biting, lifting, flicking, shaking, or gripping the injected paw, are observed. Formalin typically produces a biphasic response, with a brief, intense acute response (Phase I) occurring 0 to 10 minutes after injection, followed by a brief period of low response, and then a sustained response (Phase II) that begins approximately 10 to 15 minutes after injection, peaks, gradually declines, and usually remains at a high level for 60 minutes or longer after injection. This assay is an empirical form of non-stimuli-induced spontaneous nociceptive behavior, and the persistent nature of these behaviors is particularly relevant to the biological understanding of chronic pain. "Nociceptive" behaviors, including paw shaking, tending to injured limbs, and voluntary movements, are more complex than nociceptive behaviors and can be included in the composite score; however, they are difficult to score manually in mice.

[0038] Formalin assays rely on human observers to identify when animals exhibit nociceptive behaviors. This reliance makes observation labor-intensive, time-consuming, and highly subjective, as different nociceptive behaviors are not always uniformly defined and recorded. Rating scales are affected by inter-observer variability; some behaviors, such as tending to an injured limb or lifting a limb, have been reported to be difficult to reliably score in mice. Therefore, the labeling of mouse behavior is often limited to licking / biting behaviors.

[0039] This disclosure relates to automated phenotypic analysis of subject behavior, particularly for formalin measurements. Automated phenotypic analysis of behavior can overcome the aforementioned rating bias and inefficiencies associated with human observers. Formalin measurements can be performed via video recording, and the video can be processed using the systems and methods described herein to label subject behavior.

[0040] The systems and methods of this disclosure may be particularly useful for phenotypic analysis of the behavior of smaller subjects, such as mice, and may be able to distinguish smaller or similar movements, such as biting or licking. Smaller subjects, such as mice, tend to move faster compared to other larger subjects, such as rats. This disclosure describes techniques for using high-speed video to assess specific behaviors of subjects, as well as techniques for accurately assessing rapid withdrawal reflexes used in many nociceptive assays.

[0041] Conventional systems track one or two body parts of a subject and also use physical markers on the subject's body surface (such as bleaching the subject's fur or applying dyes to the subject's body surface). This disclosure relates to tracking multiple points on a subject's body to measure changes in distance between various body parts, thereby determining when the subject exhibits nociceptive behavior. Subject behavior is materialized as a series of movements, represented as changes in the orientation of body parts over time. For example, in a formalin assay, a mouse's licking behavior can take several different postural profiles because the mouse can bend towards its paw, raise its paw, or move its paw rapidly. In an example embodiment of this disclosure, a machine learning (ML) model can identify and track multiple body parts of the subject. Using an ML model allows for the calculation of many relative body part orientations and allows for the use of more complex manifestations of the subject's body in formalin assays.

[0042] Mice are an important component of pain and analgesia research due to their genetic susceptibility; therefore, developing automated nociceptive scoring in mice is important for large-scale studies. Recent advances in machine learning allow for accurate classification of specific subject behaviors across the full length of any recorded video. The advantages of such a scalable system include savings in time, labor, and information, and the elimination of the need for restrictive sampling methods. Improved scoring methods for formalin assays enhance reliability and reproducibility by improving the consistency of measurement results. Automated phenotypic analysis also meets the ethical requirements of substitution, reduction, and optimization. In some embodiments, this disclosure includes a supervised machine learning approach using recorded formalin assays performed on laboratory mice. The system is validated through extensive comparisons with manual scoring. To assess the applicability to the widely used C57BL / 6J-derived strain and to C57BL / 6N-derived deletion mutants widely collected by the International Mouse Phenotyping Consortium, comparisons were made between the two strains.

[0043] The system disclosed herein can provide many benefits. One benefit could be improvements in examining nociceptive behavior, thereby increasing reliability and reducing the need for repeated experiments. Another benefit could be reduced costs of conducting experiments and shorter time to generate results for such experiments.

[0044] In some implementations, the system disclosed herein includes three components / three functions for automated phenotypic analysis: (1) point detection (based on various body parts); (2) frame-level feature extraction; and (3) labeling of subject behavior. Configuring the system in this way enables it to combine changes when tracking different body parts of a subject with changes when determining different behaviors exhibited by the subject.

[0045] The system 100 disclosed herein can be used as follows: Figure 1 The various components shown operate the system. System 100 may include image capture devices 101 and 102 and one or more systems 150 connected via one or more networks 199. Image capture device 101 may be part of, included in, or connected to another device (e.g., device 1600), and may be a camera, high-speed camera, or other type of device capable of capturing images and video. In addition to or in lieu of image capture devices, device 101 may include motion detection sensors, infrared sensors, temperature sensors, atmospheric condition detection sensors, and other sensors configured to detect various characteristics / environmental conditions. Device 102 may be a laptop computer, desktop computer, tablet computer, smartphone, or other type of computing device capable of displaying data, and may include one or more components described below in conjunction with device 1600.

[0046] Image capture device 101 can capture video (or one or more images) of one or more subjects subjected to formalin testing, and can transmit video data 104 representing the video to system 150 for processing as described herein. System 150 may include Figure 1 The system 150 includes one or more components and can be configured to process video data 104 to determine the behavior of a subject over time. The system 150 can determine output label data 130 that associates one or more frames of video data 104 with labels representing the subject's behavior. The output label data 130 can be sent to device 102 to be output to a user for observation of the processing results of video data 102.

[0047] The following describes details of various components of system 150. These components may reside on the same or different physical devices. Communication between these components may occur directly or via network 199. Communication between device 101, system 150, and device 102 may occur directly or via network 199. One or more components shown as part of system 150 may be located at device 102 or at a computing device (e.g., device 1600) connected to image capture device 102.

[0048] Figure 2This is a flowchart illustrating a process 200 for analyzing video data 104 of a subject to determine the subject's behavior according to an embodiment of this disclosure. Figure 2 The steps of the demonstrated process can be performed by system 150. In other embodiments, one or more steps of the process can be performed by device 102 or a computing device associated with image capture device 101.

[0049] System 150 receives (202) video data 104 representing the movement of at least one subject. In some cases, the subject is a mouse. In some embodiments, video data 104 is a top-down camera view of the subject in an open field activity area. In some cases, video data 104 represents movement in (four) spaced enclosures (e.g., Figure 3The video shows multiple mice (e.g., four mice). The video can capture the subject's movements over a period of time (e.g., one hour, 90 minutes, two hours, etc.). The subject may have been given formalin, and the captured movements can represent the subject's physical response (i.e., behavioral response) to the effects of formalin. In some embodiments, the subject may be given other types of solutions / formulas to induce nociceptive behavior. Nociceptive behavior is the neural process of encoding and processing noxious stimuli. Nociceptive behavior refers to signals reaching the central nervous system due to stimulation of specialized sensory receptors called nociceptors in the peripheral nervous system. In some embodiments, the nociceptive stimuli used in the method of the present invention are one or more of heat, high-intensity light, pressure, cold, physical injury, electricity, and chemicals, wherein the chemicals include, but are not limited to, acetone and complete Freund's adjuvant (CFA), capsaicin, etc. In some embodiments of the invention, these and other nociceptive stimuli can be used to induce pain in the subject. In some embodiments, inducing pain includes exposing the subject to one or more of the following stimuli: heat, light, pressure, cold, electricity, and chemical agents. In some embodiments of the invention, a subject is exposed to one or more stimuli selected from heat, light, pressure, cold, electricity, and chemical agents, and these stimuli are used in a manner and amount sufficient to induce a desired level of pain and / or pain. In some embodiments, means of inducing pain in a subject include inducing inflammation in the test subject. In some embodiments, means of inducing inflammation include exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, electricity, and chemical agents. In some embodiments, the chemical agents include one or more of formalin and acetone. In some embodiments, means of inducing pain include one or more of the following: exposing the test subject to an effective amount of a pain-inducing agent, such as exposing the test subject to an effective amount of a pain-inducing chemical agent; implanting a pain-inducing agent (such as, but not limited to, stimulating electrodes, sustained-release chemicals, etc.) into or on the body surface of the subject; injecting the pain-inducing chemical agent into the subject, etc. Other methods of inducing pain known in the art are also applicable to the methods of the present invention, and the amount and level of the pain-inducing agent can be selected using means known in the art.

[0050] Video data 104 may correspond to video captured by device 101. In one example implementation, video data 104 represents 30 frames per second with 704 × 480 pixels. In some cases, video data 104 may correspond to images (image data) captured by device 101 at specific time intervals, such that these images capture the subject's movement over a period of time.

[0051] System 150 uses point tracker component 110 to process (204) video data 104 to identify point data representing multiple body parts of a subject. Point tracker component 110 can be configured to identify various body parts of the subject. These body parts can be identified using point data such that first point data can correspond to a first body part, second point data can correspond to a second body part, and so on. In some embodiments, point data can be one or more pixel positions / coordinates (x, y) corresponding to a body part. Point tracker component 110 can be configured to identify pixel positions corresponding to a specific body part within multiple frames of video data 104. Point tracker component 110 can track the movement of that specific body part during the video by identifying the corresponding pixel positions. Point data 112 can indicate the position of that specific body part during a specific frame of the video. Point data 112 provided to feature extraction 115 can include the positions of all body parts identified and tracked by point tracker component 110 across multiple frames of video data 104.

[0052] In some implementations, when the subject is a mouse, the dot tracker component 110 can identify and track the following body parts: mouth, nose, right forepaw, left forepaw, right hind paw, left hind paw, abdomen, and tail root. Figure 4A and Figure 4B It shows an example body part that is tracked using point data. Figure 4A Labels (x, y pixel coordinates) are shown for marking 12 points (mouth; nose; right forepaw; left forepaw; 3 points on each hind paw—outer, inner, and bottom; abdomen; and tail root) of the test mice. Figure 4B These 12 points for each mouse are shown; they are the outputs of the keypoint tracker, connected here by a "skeleton" and indicated by "body / head" circles for orientation. Figure 3 and Figure 9 An example of input video data is shown. Each video captures the movement of four test mice in four cages. In some embodiments, the inner walls of the cages may also be marked for identification by the point tracker component 110.

[0053] Point data 112 can be a vector, array, or matrix representing the pixel coordinates of various body parts across multiple video frames. For example, point data 112 can be [Frame 1 = {Mouth:(x1,y1); Hind paw:(x2,y2)}], [Frame 2 = {Mouth:(x3,y3); Fore paw:(x4,y4)}], and so on. In some implementations, the point data 112 for each frame can include at least 12 pixel coordinates representing 12 parts / body parts of the subject to be tracked by the point tracker component 110.

[0054] System 150 uses feature extraction component 115 to process (206) point data 112 to determine feature vectors representing at least distance and angle features. Feature extraction component 115 can determine distances between various body parts of the subject and generate one or more distance feature vectors 118. Feature extraction component 115 can determine first distance data between two (first pair) body parts, second distance data between another two (second pair) body parts, and so on, for multiple video frames. Figure 7A An example distance feature vector is shown, which includes distance data representing the distance between two specific body parts across multiple video frames. The feature extraction unit 115 can determine a first distance feature vector representing the distance between a first pair of body parts across multiple video frames, a second distance feature vector representing the distance between a second pair of body parts across multiple video frames, and so on.

[0055] The feature extraction unit 115 can determine the angles between various body parts of the subject and generate one or more angle feature vectors 116. The feature extraction unit 115 can determine first angle data between three (first triplet) body parts, second angle data between another three (second triplet) body parts, and so on, for multiple video frames. Figure 7B An example angular feature vector is shown, which includes angular data representing the angles between three specific body parts across multiple video frames. The feature extraction unit 115 can determine a first angular feature vector representing the angles between a first set of body parts across multiple video frames, a second angular feature vector representing the angles between a second set of body parts across multiple video frames, and so on.

[0056] System 150 uses behavior classification component 120 to process (208) feature vectors to determine subject behavior. In some embodiments, behavior classification component 120 may process distance feature vector 118, angular feature vector 116, or both. Behavior classification component 120 may use one or more trained machine learning (ML) models to process these feature vectors. In some embodiments, the ML model may be a classifier configured to process distance and / or angular features to determine whether a subject exhibits a specific behavior based on the position of one or more body parts relative to other body parts. For example, in some cases, a test mouse may exhibit nociceptive behavior by licking or biting its paws (e.g., hind paws). Behavior classification component 120 may process distance and / or angular features to determine whether the mouse's paws are near or at the mouth of the mouse indicating the licking or biting behavior. Behavior classification component 120 may label each video frame with a specific behavior (e.g., licking, biting, no biting, no licking, etc.).

[0057] In some implementations, the behavior classification component 120 may use an ML model to determine the probability (or score, confidence score, etc.) corresponding to the likelihood that a subject will exhibit a specific behavior during the video frame. In this case, the output of the ML model may be a number between 0 and 1, or between 0 and 100, or any other such numerical range. The behavior classification component 120 may further process the output of the ML model to determine the output label 130 corresponding to the video frame of the video data 104.

[0058] In some implementations, the behavior classification component 120 can be configured to perform binary classification, where a particular video frame can be classified into one of two behavior categories / types (e.g., noxious behavior or noxious behavior; biting or no biting; licking or no licking; etc.). In this case, the behavior classification component 120 can associate values ​​such as false or true, yes or no, 1 or 0 with the video frame indicating whether the subject exhibited a specific behavior during that video frame.

[0059] In some implementations, the behavior classification component 120 can be configured to perform multi-class or multiple-category classification, wherein a particular video frame can be classified into one of three or more behavior categories / types (e.g., no harm, licking, or biting). In some implementations, the behavior classification component 120 can be configured to perform multi-label classification, wherein a particular video frame can be associated with more than one behavior category / type.

[0060] The behavior classification component 120 may include one or more machine learning (ML) models, including but not limited to one or more classifiers, one or more neural networks, one or more probabilistic graphs, one or more decision trees, etc. In other embodiments, the behavior classification component 120 may include a rule-based engine, one or more statistical algorithms, one or more mapping functions, or other types of functions / algorithms to determine whether specific distance features and / or angular features indicate a specific behavior.

[0061] Although the examples describe the use of automated phenotypic analysis methods and systems on test mice, it should be understood that the systems and methods described herein can be configured to perform automated phenotypic analysis on other types of subjects, such as rats, rabbits, hamsters, etc. Additional information regarding subjects for whom the methods and systems of this invention can be used is provided elsewhere herein.

[0062] In some embodiments, the point tracker component 110 may implement one or more trained ML models configured to identify and track various body parts of a specific type of subject. The ML model may be a neural network (e.g., a deep neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), etc.). In other embodiments, the ML model of the point tracker component 110 may be other types of ML models. The ML model of the point tracker component 110 may be configured for 3D label-free pose estimation based on transfer learning utilizing deep neural networks.

[0063] In one example implementation, the training data for configuring the point tracker component 110 can be video frames from various videos of mice that have been administered formalin. These video frames can include the movement of the mice over a period of time (e.g., 90 minutes). In some implementations, these video frames can be from videos of mice that have not yet been administered formalin. Various body parts of the mice can be labeled in the training data. For example, the video frames can be as follows: Figure 4A and Figure 4B The display includes 12 markers corresponding to the following body parts: mouth, nose, right forepaw, left forepaw, three points on each hind paw (outer, inner, and bottom), mid-abdomen, and tail root. If the point tracker component 110 is configured to track multiple mice within the input video data, the training data can be videos of multiple mice separated by different cages, and the training data can include labels identifying the cage walls. The training data can also include video frames corresponding to empty cages. Configuring the point tracker component 110 to track multiple mice within the input video data eliminates the need for cropping or manipulating the input video data and enables the system to perform phenotypic analysis on multiple mice simultaneously.

[0064] The point data 112 received by the feature extraction unit 115 may include (x,y) pixel coordinates of each specified body part location, and a probability estimate of the consistency of a fractional map indicating the probability of a particular body part at the corresponding pixel. Figure 5 and Figure 6This is a flowchart illustrating the process (500, 600) of the feature extraction unit 115 determining feature vectors. For each video frame, the feature extraction unit 115 determines (502) the distance between a pair of body parts of the subject, where point data 112 is used to identify the body parts. For example, using 12 points on the subject's body, the feature extraction unit 115 can determine 66 pairs of body parts. In some embodiments, the feature extraction unit 115 can determine the Euclidean distance between the body parts in this pair. For each video frame, the feature extraction unit 115 determines (602) the angle between triplets of the subject's body parts, where point data 112 is used to identify the body parts. For example, using 12 points on the subject's body, the feature extraction unit 115 can determine 15 triplets of body parts. The feature extraction unit 115 can use the midpoint of this triplet of body parts to determine the angle.

[0065] The distance and angle determined at steps 502 and 602 represent the relative body part position information for a single frame. The automated phenotyping system can be configured to detect changes in the relative orientation of body parts over time and identify these changes as actions / movements performed by the subject. The automated phenotyping system can be configured to identify behaviors based on a series of actions / movements performed by the subject. In this regard, the feature extraction unit 115 can use consecutive video frames to observe changes in distance and angle over a period of time.

[0066] The feature extraction unit 115 selects (504) frames of interest and performs (506) statistical calculations using distance data from the frame windows surrounding those frames. Based on these statistical calculations, the feature extraction unit 115 generates (508) distance feature vectors. The frame extraction unit 115 selects (604) frames of interest and performs (606) statistical calculations using angle data from the frame windows surrounding those frames. Based on these statistical calculations, the feature extraction unit 115 generates (608) angle feature vectors.

[0067] In some implementations, the feature extraction unit 115 may use a window with six frames to select two video frames before the frame of interest and three video frames after the frame of interest. In other implementations, the feature extraction unit 115 may use a window with eleven frames to select five frames before the frame of interest and five frames after the frame of interest. In still other implementations, the feature extraction unit 115 may use a window with twenty-one frames to select ten frames before the frame of interest and ten frames after the frame of interest. Figure 7A and Figure 7BExample feature vectors are shown over 24 consecutive frames, consisting of a pair of distances from the point representing the body part “Left Front” (LF) to the point representing the body part “Right Back Lateral” (RHout), and an angle between the points representing the body parts “Right Back Lateral” (RHout), abdomen, and “Left Back Lateral” (LHout), with the frames of interest marked at the center. Figure 7A and Figure 7B The frame of interest 710 is shown, as well as various frame windows that the feature extraction unit 115 can use. Figure 7A and Figure 7B The relative position measurements calculated for body parts for each video frame are shown (66 pairs of distances and 15 angles; only one of each feature vector is shown in the figure). Frames of assumed interest are highlighted, and measurements for preceding and following frames are shown (24 consecutive frames). Statistical inputs can be computed on windows of three different sizes (6, 11, and 21 frames), and these per-frame features can be used as input to the behavior classification component 120.

[0068] The parameter values ​​for each frame are calculated by moving the frame window and selecting frames of interest accordingly. The parameter values ​​for the frames of interest can be calculated by performing statistical analysis on the distance and angle values ​​of those frames within the window. In one example implementation, the statistical analysis may include calculating the mean, standard deviation, median, and / or median absolute deviation using the distance and angle values ​​of those frames within the window. The determined parameter values ​​are stored in the feature vector of the corresponding frame. In some implementations, the feature extraction component 115 may use a GentleBoost classifier to determine the feature vector. In some implementations, point data 112 indicating the probability estimates of the frames of interest may also be used to determine the feature vector.

[0069] A distance feature vector 118 and an angle feature vector 116 may correspond to a first time period of the length of video data 104, and another distance feature vector 118 and another angle feature vector 116 may correspond to a second time period of the length of video data 104. Each value within the distance feature vector 118 and the angle feature vector 116 may correspond to a video frame during the corresponding time period (e.g., approximately 30 milliseconds). The first value in the distance feature vector 118 and the first value in the angle feature vector 116 may correspond to the same first video frame.

[0070] In some implementations, the feature extraction unit 115 can determine feature data representing features corresponding to the video data 104 at the frame level, and the feature data can be represented in a form different from the feature vector.

[0071] One or more ML models for the behavior classification component 120 can be configured using training data, which may include labeled / annotated video data. The training video data can be labeled to indicate when a subject's movements (captured in the video) correspond to a specific behavior. The training video data may include a first label associated with a video frame (e.g., a 30-millisecond duration of the video) and a second label associated with another video frame, where the first label indicates that the subject begins to exhibit a specific behavior that the automated phenotyping system is configured to detect, and the second label indicates that the subject stops exhibiting that specific behavior.

[0072] The configuration of the training video data can depend on the configuration of the automated phenotyping system. When the behavior classification component 120 is configured to classify a subject's behavior based on whether the subject exhibits a certain behavior, the training video data may include labels associated with video frames in which the subject exhibits the behavior. For example, when the automated phenotyping system is configured to detect licking behavior in mice, the training video data may include labels associated with video frames in which the mouse licks its hind paws during that period. When the behavior classification component 120 is configured to classify a subject's behavior based on the type of behavior exhibited by the subject, the training video data may include labels for a first type, labels for a second type, etc., where the first type of label is associated with video frames in which the subject exhibits the first type of behavior during that period, and the second type of label is associated with video frames in which the subject exhibits the second type of behavior during that period.

[0073] In some embodiments, the behavior classification component 120 can be configured to detect specific movements made by the subject to indicate a particular behavior. For example, the behavior classification component 120 can be configured to detect a mouse licking its right hind paw to indicate a licking behavior (where formalin was injected into the right hind paw). In other embodiments, the behavior classification component 120 can be configured to detect different types of movements made by the subject to indicate a specific behavior. For example, the behavior classification component 120 can be configured to detect a mouse licking its right or left hind paw to indicate a licking behavior. In some embodiments, any contact between the hind paw and the mouth (e.g., mouth touching the hind paw, hind paw approaching the mouth, mouth licking the hind paw, mouth biting the hind paw, etc.) can be labeled as a nociceptive behavior.

[0074] Figure 8This is a flowchart of a process 800 that can be performed by the behavior classification component 120. The behavior classification component 120 receives (802) feature vectors (e.g., 116, 118) or feature data determined by the feature extraction component 115. The behavior classification component 120 processes (804) these feature vectors to determine a label for each video frame in the video data 104. Based on the system configuration, the label can indicate the type of behavior exhibited by the subject during that video frame. Because these feature vectors include values ​​determined through statistical analysis using features of frames surrounding the frame of interest, they provide the behavior classification component 120 with context about the subject's movement before and after a particular frame of interest. The behavior classification component 120 can use one or more ML models to process the feature vectors, and the output of the ML model can be a score / probability indicating the likelihood of each video frame corresponding to a specific behavior, or it can be a probability distribution / vector of the scores for each video frame. For example, in the case where the behavior classification component 120 is configured to determine whether a subject exhibits a certain behavior, a first video frame can be associated with a first score, a second video frame can be associated with a second score, and so on. In another instance, where the behavior classification component 120 is configured to determine whether a subject exhibits one of two behaviors, a first video frame may be associated with a score vector {score1, score2}, where score1 indicates the likelihood that the subject exhibits the first behavior and score2 indicates the likelihood that the subject exhibits the second behavior. In some implementations, the ML model outputs a label (e.g., true or false, yes or no, 0 or 1, etc.) for each video frame to indicate whether the subject exhibits a certain behavior.

[0075] The behavior classification component 120 determines (806) output labels based on binning the output of the ML model, and then sends (808) these output labels to the device 102 for display to the user. The behavior classification component 120 can be configured to bin the output of the ML model based on a fixed number of video frames, duration, or the ML model output. For example, the behavior classification component 120 can bin the ML model output into bins of 5 minutes in length, where the labels associated with the 5-minute bins are determined using the scores / labels corresponding to the video frames included in the 5-minute bins. The bin size can depend on the type of information required by the user, the type of subject, the type of behavior to be detected, and other factors.

[0076] An automated phenotyping system can be configured to process single-video capture motion of multiple subjects. In this case, a point tracker component 110 is configured to identify multiple subjects, identify and track body parts of each subject, and generate point data 112 corresponding to each subject. The point tracker component 110 can generate first point data 112 corresponding to a first subject, second point data 112 corresponding to a second subject, and so on. A feature extraction component 115 is configured to process the point data 112 and generate a feature vector corresponding to each subject. The feature extraction component 115 can generate a first feature vector corresponding to a first subject, a second feature vector corresponding to a second subject, and so on. A behavior classification component 120 is configured to process the feature vectors corresponding to multiple subjects and generate output labels indicating when / whether a subject exhibits a specific behavior.

[0077] The automated phenotypic analysis system described herein may be an effective way to score / classify subject behaviors across various assays. The classification performed by the system described herein is comparable in some respects to manual classification of subject behaviors. Furthermore, the automated phenotypic analysis system can be used to process the behaviors of multiple subjects simultaneously, enabling scalability and processing of different genetic strains.

[0078] Although automated phenotyping systems are described as comprising point tracker components, feature extraction components, and behavior classification components, it should be understood that the functionality described herein can be performed using fewer or more components, or components of different types (belonging to different types of ML models or techniques). For example, different methods can be used to track body parts in video data, different window sizes can be used to handle distance and angular features, different types of ML models can be used to handle feature vectors, and so on.

[0079] In some respects, the localization of individual body parts can lead to a reduction in the amount of data to be processed (e.g., from 337,920 pixels per video frame to 159 pixels of interest (x,y coordinates and probabilities for 53 points)), making the task of tracking body parts over long-duration video manageable. Measuring the orientation of body parts relative to each other using distance and angle, as described in this paper, can be an efficient way to generate body part information for each video frame. The average localization error of the body points of interest can be less than five pixels. Because behavior classification is performed over thousands of frames (e.g., 153,000 frames over 85 minutes), even if a few frames contain large errors (regarding body part identification), their impact on the overall behavior classification is negligible. The system described in this paper uses time windows to evaluate changes over time to reduce the impact of individual frames and help smooth inconsistencies between consecutive frames.

[0080] In some embodiments, the behavior classification component 120 may be configured to detect subject behaviors involving licking or biting a body part (e.g., paws). In such embodiments, the behavior classification component 120 may employ one or more ML models trained to detect licking or biting behaviors. In other embodiments, the behavior classification component 120 may be configured to detect subject behaviors involving shaking a body part (e.g., paws). In such embodiments, the behavior classification component 120 may employ one or more ML models trained to detect shaking behaviors. In still other embodiments, the behavior classification component 120 may be configured to detect subject behaviors involving upright movements supported by the walls of an activity space in which the subject is located. In such embodiments, the behavior classification component 120 may employ one or more ML models trained to detect upright behaviors.

[0081] In other embodiments, the behavior classification component 120 can be configured to detect different types of behavior: licking or biting behavior, shaking behavior, and uprighting behavior. In such embodiments, the behavior classification component 120 can employ one or more ML models trained to detect licking or biting behavior, one or more separate ML models trained to detect shaking behavior, and one or more separate ML models trained to detect uprighting behavior. In other such embodiments, the behavior classification component 120 can employ one or more ML models trained to detect all of the following different behaviors: licking or biting behavior, shaking behavior, and uprighting behavior.

[0082] In some implementations, system 150 may be configured to determine metrics related to the subject's behavior. For example, system 150 may determine the number of times the subject exhibited licking or biting behavior during the video, the amount of time the subject exhibited licking or biting behavior during the video, and the duration of each licking or biting behavior episode. As another example, system 150 may determine the number of times the subject exhibited shaking behavior during the video, the amount of time the subject exhibited shaking behavior during the video, and the duration of each shaking behavior episode. As yet another example, system 150 may determine the number of times the subject exhibited uprighting behavior during the video, the amount of time the subject exhibited uprighting behavior during the video, and the duration of each uprighting behavior episode.

[0083] In some implementations, system 150 may be configured to determine various per-video-frame features corresponding to a subject. Such per-video-frame features may include, but are not limited to, open field metrics, gait attitude metrics, and other heuristic metrics. Behavioral classification component 120 may use one or more of the aforementioned features to detect subject behavior. In other implementations, the aforementioned features may not be considered by behavioral classification component 120, but may instead be used to compare a subject (e.g., having a first genotype or phenotype) with another subject (e.g., having a different second genotype or phenotype).

[0084] Open field measures can correspond to subject movement within an open field activity area and may include, but are not limited to: the time spent by the subject at / near the center of the activity area, the time spent by the subject at / near the periphery of the activity area, the time spent by the subject at / near a corner of the activity area, the distance between the subject's position and the center of the activity area, the distance between the subject's position and the periphery of the activity area, the distance between the subject's position and a corner of the activity area, the number of times the subject grooms themselves, and the amount of time the subject spends grooming themselves. Such open field measures can be determined using point data 112.

[0085] Gait attitude metrics can correspond to a subject's gait / walking and may include, but are not limited to, stride width, stride length, stride length, speed, angular velocity, limb duty cycle (e.g., based on the time of standing / the amount of time the paws are in contact with the ground during stride intervals, and the total time of stride intervals), lateral displacement of the nose (given to the subject's central spine vector), lateral displacement of the tail root (given to the subject's central spine vector), lateral displacement of the tail tip (given to the subject's central spine vector), and temporal symmetry (e.g., the similarity of gait features over a set of video frames). Such open field metrics can be determined using point data 112.

[0086] Other heuristic features may correspond to other movements of the subject and may include, but are not limited to: the amount of time the subject is frozen / immobile, the number of times the subject is frozen, the number of times the subject has continuous activity / gait, and the amount of time the subject has continuous activity / gait. Such heuristic features can be determined using point data 112, open field measures, and / or gait attitude measures.

[0087] spliced ​​video feed

[0088] In some implementations, video data 104 can be generated using multiple video feeds capturing subject movement from multiple different angles / viewpoints. Video data 104 can be generated by stitching / combining a first video of a subject's top view and a second video of a subject's side view. The first video can be captured using a first image capture device (e.g., device 101a), and the second video can be captured using a second image capture device (e.g., device 101b). Other views of the subject may include a right-side view, a left-side view, a top-down view, a bottom-up view, a front-side view, a rear-side view, and other views. Videos from these different views can be combined to generate video data 104, providing a comprehensive / expanded view of the subject's movement, which can lead to more accurate and / or more effective classification of subject behavior by an automated phenotypic analysis system. In some implementations, videos from different views can be combined to provide a wide field of view with a short focal length while preserving a top-down perspective across the entire view. In some implementations, multiple videos from different views can be processed using one or more ML models (e.g., neural networks) to generate video data 104. In some implementations, the system can use 2D video / images to generate 3D video data.

[0089] In some implementations, various techniques can be used to synchronize video captured by multiple image capture devices 101. For example, the multiple image capture devices 101 can be synchronized with a central clock system and controlled by a master node. Synchronizing multiple video feeds can involve using various hardware and software, such as adapters, multiplexers, USB connections between image capture devices, wireless or wired connections to network 199, software for controlling the devices (e.g., MotionEyeOS), etc.

[0090] In one example implementation, image capture device 101 may be an ultra-wide-angle lens (i.e., a fisheye lens), which produces strong visual distortion and is designed to create wide panoramic or hemispherical images, and is capable of achieving extremely wide angles of view. In one example implementation, the system for capturing video data 104 may include four fisheye lenses connected to four single-board computing devices (e.g., Raspberry Pi), and additional image capture devices for capturing top-down views. The system may use various techniques to synchronize these components. One technique involves pixel / spatial interpolation, for example, where, in the case that a point of interest (e.g., a body part on a subject) is located at (x, y), the system relative to time identifies the orientation along the x and y axes within the top-down view video. In one instance, the pixel interpolation along the x-axis may be calculated by the single-board computing device according to the following equation:

[0091] (Pi offset ΔX / Pi offset ΔT) * (Top-down view offset ΔT) + Initial point (x)

[0092] The equation can then be used to calculate the orientation of the point of interest along the y-axis. In some implementations, to address lens distortion during video calibration, padding can be added to one or more video feeds (instead of scaling the video feeds).

[0093] Subjects

[0094] Some aspects of this invention include the use of automated phenotypic analysis methods on subjects. As used herein, the term "subject" can refer to a human, non-human primate, cattle, horse, pig, sheep, goat, dog, cat, bird, rodent, or other suitable vertebrate or invertebrate organism. In some embodiments of the invention, the subject is a mammal, and in some embodiments of the invention, the subject is a human. In some embodiments, the methods of the invention can be used on rodents, including but not limited to mice, rats, gerbils, hamsters, etc. In some embodiments of the invention, the subject is a normal, healthy subject, while in some embodiments, the subject is known to have, at risk to have, or suspected of having a disease or condition. The terms "subject" and "test subject" are used interchangeably herein.

[0095] As a non-limiting example, the subjects evaluated using the automated phenotypic analysis method of the present invention may be subjects serving as animal models of pain diseases or conditions, such as models of one or more of the following: inflammatory pain, neuropathic pain, myalgia, arthralgia, chronic pain, visceral pain, cancer pain, and postoperative pain. Additional chronic pain models suitable for the methods and systems of the present invention are known in the art, see, for example: Barrot M. Neuroscience 2012, 211:39-50; Graham, DM, Lab Anim (NY) 2016, 45:99-101; Sewell, RDE, Ann Transl Med 2018, 6:S42.2019 / 01 / 08; and Jourdan, D. et al., Pharmacol Res 2001, 43:103-110, the contents of which are incorporated herein by reference in their entirety.

[0096] In some embodiments, the automated phenotyping method or system of the present invention can be used to monitor subjects in which pain is not induced. For example, externally induced pain-inducing actions, such as injection of chemical agents or exposure to pain-inducing heat, light, pressure, etc., can be avoided when subjecting a test subject serving as a model of a pain condition. In some embodiments of the present invention, the test subject is an animal model of neuropathic pain, and the test subject is monitored using the automated phenotyping method and / or system of the present invention without inducing additional pain in the test subject. In some embodiments of the present invention, the test subject serving as an animal model of a pain condition can be used to evaluate the test subject's response to the pain condition. Furthermore, candidate therapeutic agents or methods can be administered to the test subject serving as an animal model of a pain condition, and the test subject can be monitored using the automated phenotyping method or system of the present invention, and the efficacy of the candidate therapeutic agent or method in alleviating the pain of the test subject's pain condition can be determined.

[0097] In some embodiments of the automated phenotypic analysis method of the present invention, the subjects are wild-type subjects. As used herein, the term "wild-type" refers to the typical form of phenotype and / or genotype of a species as it appears in nature. In some embodiments of the present invention, the subjects are non-wild-type subjects, for example, subjects with one or more genetic modifications compared to the wild-type genotype and / or phenotype of the subject species. In some cases, the genotypic / phenotype difference between the subject and the wild-type is caused by hereditary (germline) mutations or acquired (somatic) mutations. Factors that may cause subjects to exhibit one or more somatic mutations include, but are not limited to: environmental factors, toxins, ultraviolet radiation, spontaneous errors in cell division, teratogenic events such as, but not limited to, radiation, maternal infection, chemicals, etc.

[0098] In some embodiments of the method of the present invention, the subject is a genetically modified organism, also known as an engineered subject. An engineered subject may include pre-selected and / or intentionally genetic modifications, thus exhibiting one or more genotypic traits and / or phenotypic traits different from those of a non-engineered subject. In some embodiments of the present invention, conventional genetic engineering techniques may be used to generate engineered subjects that exhibit genotypic and / or phenotypic differences compared to non-engineered subjects of the same species. As a non-limiting example, genetically engineered mice and the method or system of the present invention can be used to evaluate the phenotype of the genetically engineered mice, and the results can be compared with results obtained from a control (control results), wherein the functional gene product is absent or present at a reduced level in the genetically engineered mouse.

[0099] Testing and screening controls and candidate compounds

[0100] Results obtained from subjects using the automated phenotypic analysis method or system of the present invention can be compared with control results. The method of the present invention can also be used to assess phenotypic differences in subjects relative to controls. Therefore, some aspects of the present invention provide a method for determining whether there is a change in activity in a subject compared to a control. Some embodiments of the present invention include using the automated phenotypic analysis method of the present invention to identify phenotypic features of a disease or condition, and in some embodiments of the present invention, automated phenotypic analysis is used to assess the effects of candidate therapeutic compounds on subjects.

[0101] Results obtained using the automated phenotypic analysis method or system of the present invention can be advantageously compared with controls. In some embodiments of the invention, the automated phenotypic analysis method may be used to evaluate one or more subjects, followed by retesting of these subjects after administration of a candidate therapeutic compound. The terms “subject” and “test subject” are used herein to refer to a subject evaluated using the method or system of the present invention, and the terms “subject” and “test subject” are used interchangeably herein. In some embodiments of the invention, results obtained from evaluating test subjects using the automated phenotypic analysis method are compared with results obtained from performing automated phenotypic analysis on other test subjects. In some embodiments of the invention, results for test subjects are compared with results from automated phenotypic analyses performed on the test subject at different times. In some embodiments of the invention, results obtained from evaluating test subjects using the automated phenotypic analysis method are compared with control results.

[0102] As used herein, the control result can be a predetermined value, which can take various forms. It can be a single cutoff value, such as the median or mean. It can be established based on comparison groups, such as subjects who have already been evaluated using the automated phenotypic analysis system or method of the present invention under similar conditions to the test subjects, wherein the test subjects were administered the candidate treatment agent, while the comparison groups were not exposed to the candidate treatment agent. Another example of a comparison group can include subjects known to have a disease or condition and a group without the disease or condition. Another comparison group can be subjects with a family history of the disease or condition and subjects from a group without such a family history. For example, predetermined values ​​can be set where the tested population is divided equally (or unequally) into multiple groups based on the test results. Those skilled in the art can select appropriate control groups and control values ​​for the comparison method of the present invention.

[0103] The automated phenotyping method or system of the present invention can monitor whether subjects undergo changes under test conditions relative to control conditions. As a non-limiting example, changes occurring in subjects may include, but are not limited to, one or more of the following: movement frequency, licking behavior, response to external stimuli, etc. The methods and systems of the present invention can be used to test subjects to assess the impact of their disease or condition, and also to evaluate the efficacy of candidate therapeutics. As a non-limiting example of using the method of the present invention to assess the presence of changes in test subjects as a means of identifying the efficacy of candidate therapeutics, the automated phenotyping method of the present invention is used to assess test subjects known to have a pain condition. A candidate therapeutic is then administered to the test subject, and the automated phenotyping method is used again for evaluation. The presence or absence of changes in the test subject's results accordingly indicates whether the candidate therapeutic has an effect on the pain condition.

[0104] It should be understood that, in some embodiments of the present invention, test subjects can act as their own controls, for example, by evaluating the subject two or more times using the automated phenotypic analysis method of the present invention, and then comparing the results obtained from these two or more different evaluations. The methods and systems of the present invention can be used to assess the progression or regression of a subject's disease or condition by performing two or more evaluations on the subject using one embodiment of the methods or systems of the present invention, thereby identifying and comparing changes in phenotypic characteristics in the subject over time.

[0105] Example

[0106] Example 1. Model Development: Data Training, Testing, and Model Validation

[0107] method

[0108] Animal care

[0109] Mice were single-sex and housed in groups of 3 to 5 mice under a 12-hour light-dark schedule, with free access to water and food. Experiments were conducted during the light phase. The model was trained, tested, and validated using video data from 166 mice (Jackson Laboratories, C57BL / 6NJ=JR005304: males n=53, females n=37; C57BL / 6J=JR000664: males n=46, females n=30). Mice (11 to 17 weeks old) were tested over 25 experimental periods; at the end of each period, all mice were euthanized by cervical dislocation. All procedures and protocols were approved by the Animal Care and Use Committee of the Jackson Laboratory and performed in accordance with the National Institutes of Health's guidelines for laboratory animal care and use.

[0110] Video data acquisition

[0111] Video data of mouse behavior in response to formalin injection in the hind paw was collected for training, testing, and validation of an automated phenotypic analysis system. A transparent acrylic cage (22cm long × 21.6cm wide × 12.7cm high; IITC LifeScience, Woodland Hills, CA) was placed on a transparent glass surface. This cage contained four test activity areas separated by opaque black walls (as shown in Figure 4 and...). Figure 9 (As shown). A black-and-white Dinion camera (Bosch, Farmington Hills, MI) was placed directly below the glass floor of the enclosure (16 cm) to provide the best view of the paws, and recording began from an empty enclosure under the control of Noldus Media Recorder v4 software (Noldus, Leesburg, VA). Each of the four enclosures was equipped with a dedicated camera, allowing a total of 16 mice to be tested simultaneously. There were lighting differences between the four enclosures, but the lighting was optimized to reduce glare and reflections by adding a white polycarbonate cover (23.5 cm long × 12.1 cm wide × 1 cm high; internally manufactured) to the top of each enclosure. 90 minutes of video (30 frames per second: 704 pixels × 480 pixels) was recorded after the last mouse entered the activity area. The video was extended to more than 60 minutes to ensure that any strain differences during peak behavior were captured.

[0112] Formalin was administered to mice under anesthesia to maximize consistency between injection site and delivery volume, and to minimize stress. Under gas anesthesia (4% isoflurane; Henry Schein Isothesia, Dublin, OH), 30 μl of 2.5% formalin saline solution [formaldehyde solution (Sigma-Aldrich, St. Louis, MO); sterile saline solution (Henry Schein, Dublin, OH)] was injected (intrapotomy) into the right hind paw of the mouse. The mouse was then transferred to the first test activity area, and the procedure was repeated for the next three mice before placing them in the same enclosure. Typically, the mice regained consciousness from anesthesia within one minute of being placed in the test activity area and were mobile within three minutes.

[0113] (1) Training data for point detection

[0114] To create a training set for point detection (point tracker component 110), frames were pseudo-randomly selected from eight mouse videos covering four different cages and ensuring presentation of early (up to 30 minutes), mid (30 to 60 minutes), and late (60 to 90 minutes) segments of the recording. Labels were manually applied to the desired points across 370 frames. Figure 4A and Figure 4B As shown, each mouse was marked with 12 dots (mouth, nose, right forepaw, left forepaw, 3 dots on each hind paw (outer, inner, and bottom), mid-abdomen, and tail root), and the inner wall of each activity area was marked with 5 dots (e.g., Figure 4A (As shown). Therefore, a total of 53 points were labeled for each frame. The point tracker was trained to find all 53 points for each frame, so it was not necessary to crop or manipulate video frames to locate individual activity areas. For training purposes, the positions of the grid walls were included to verify that all 12 points of the mouse were located within a single activity area. Any missing or blurred points were marked as positions x=0, y=0, and the accuracy of all labeled frames was rechecked visually. Instances of empty activity areas were included in the training. To increase the number of frames used for training, these 370 frames were reflected and rotated so that each mouse appeared in each of the four positions, for a total of 1480 labeled frames. To increase the variability of lighting conditions used for training, approximately 11% of these 1480 frames were enhanced by adding Gaussian noise (40 frames) or contrast variation (39 frames), brightness filtering (39 frames), or gamma filtering (40 frames) (Table 1 below). These enhanced frames were pseudo-randomly selected and evenly distributed across the initial 370 frames, as well as each of the reflection and rotation conditions. After these adjustments, the set of labeled frames was randomly split into a training set (85%) and a test set (15%) for validation.

[0115] Table 1. Approximately 11% of the images were adjusted.

[0116]

[0117]

[0118] (1)(a) Attitude estimation

[0119] The point tracker component 110 can utilize a pre-trained residual network (ResNet50) for body part detection. The residual network architecture uses convolutional layers to learn specific visual features, and the skip function minimizes information loss, thereby enhancing the extraction of global rules.

[0120] Tensorflow™ was used to train the ResNet50 architecture on a Tesla P100 GPU (Nvidia, Santa Clara, CA). The model with point tracker component 110 was trained for 750,000 iterations, achieving a training accuracy of 1.9 pixels and a test error of approximately 4.4 pixels on all test frames and test points. Figure 9An example of a single test frame is shown (average error of 2.4 pixels): in activity location 4, the right front paw missed 4.3 pixels, which is an approximation of the average error across all test frames. Performance stability was verified by repeating the training with different training and test sets (training error 1.9, test error 4.3).

[0121] The trained model was locked and then used to track experimental videos. These videos were approximately 100 to 120 minutes long (ranging from 1.6 GB to 2.2 GB); each frame contained 337,920 pixels (704 × 480), and the rate at which 53 points were marked varied between 36 and 37 frames per second (on a Tesla GPU). Tracking four mice in the videos was actually slightly faster than recording the videos at 30 frames per second.

[0122] (2) Frame feature extraction

[0123] Feature vectors are generated using feature extraction component 115 and the (x,y) pixel coordinates of each specified body part, along with a probability estimate based on a consistent fractional map indicating the probability of that body part at that pixel. When the activity area is empty, all 12 points are located with extremely low probabilities (e.g., >0.0001), but once a mouse is placed in the activity area, the probability estimates for all points increase. An average probability threshold of 0.8 is set for these 12 points to indicate the presence of a mouse.

[0124] Because the number of mice in each cage varied (from 1 to 4), each mouse was categorized individually. Twelve key points of interest (i.e., ...) were used. Figure 4A and Figure 4B (As shown) to generate pairwise Euclidean distances between body parts (66 pairs) and angles between selected body part triples (15 angles, as shown in Table 2 below). Figure 7A and Figure 7B Example feature vectors are shown over 24 consecutive frames, consisting of a pair of distances from the point representing the body part “Left Front” (LF) to the point representing the body part “Right Back Lateral” (RHout), and an angle between the points representing the body parts “Right Back Lateral” (RHout), abdomen, and “Left Back Lateral” (LHout), with the frames of interest marked at the center.

[0125] Table 2. Angles calculated between three body part points, where the angle is spread out with the midpoint as the vertex.

[0126] nose RHout LHout nose Right rear bottom (RHbase) Left rear bottom (LHbase) nose Right posteromedial (RHin) Left posteromedial (LHin) Mouth RHout LHout Mouth RHbase LHbase Mouth RHin LHin tail root RHout LHout tail root RHbase LHbase tail root RHin LHin abdomen RHout LHout abdomen RHbase LHbase abdomen RHin LHin LF RHout Right anterior (RF) LF RHbase RF LF RHin RF

[0127] To test the automated phenotypic analysis system, frame window sizes of 6, 11, and 21 frames (200ms, 367ms, and 700ms) were selected for the licking behavior. Each window moved along a vector, and statistical values ​​of the parameters were calculated within that time frame. A total of 1047 distinct metrics were calculated for each video frame and used as input data for the behavior classification component 120. The statistical metrics were the mean, standard deviation, median, and median absolute deviation for each distance pair. A second measurement of distance was included, which was reported as NaN if the probability of the mouth or nose dropped below 0.1, and the mean of this value and the angle was calculated. Twelve probability estimates for the frames of interest were also included as input without windowing.

[0128] (3) Behavioral Classification

[0129] For behavior classification component 120, training data was taken from 50 different videos to cover all cages, activity areas, and all sizes and sexes of mice. A total of 9300 frames were used for training, with each video no longer than 10 seconds (approximately 300 frames). The video data was annotated on a frame-by-frame basis to indicate the starting and offset frames of licking behavior. In this case, there was no distinction between licking and biting behavior, and any contact between the mouth and the right hind paw was labeled as licking. To obtain a well-balanced training set, stratified random sampling was used to select frames from clearly licking video clips (22%) and non-licking video clips (78%). The bias towards non-licking behavior was intentional because this behavior does not occur equally in the input runtime videos.

[0130] (3)(a) GentleBoost classifier model

[0131] GentleBoost (Gentle Adaptive Boosting) is an ensemble supervised learner based on minimizing exponential loss using decision trees. The GentleBoost algorithm is well-suited for binary classification responses. The classifier uses 30 weighted learners, each with up to 20 splits and a learning rate of 0.1, to fit the regression model to the predictor and label. Five-fold cross-validation is used to limit overlearning and provide estimates for the training. Due to significant redundancy in the large number of inputs, the GentleBoost model is also trained by performing Principal Component Analysis (PCA), which accounts for 99% (65 inputs) or 95% (11 inputs) of the variation.

[0132] result

[0133] Testing the GentleBoost classifier model

[0134] Table 3 shows the results for all tested classifiers for the following: precision (the proportion of frames correctly classified as licking), recall rate (the proportion of correctly identified licking frames), false positive rate (the proportion of frames incorrectly identified as licking), and overall accuracy. High values ​​in the precision-recall rate dimension indicate that the model can correctly identify licking without missing the occurrence of the behavior, regardless of how infrequently it occurs; this is particularly useful when the two behaviors are imbalanced. A low false positive rate indicates that the model does not report the behavior when licking is not present; this suggests that the model does not label all behaviors as licking to avoid missing it. The GentleBoost model performed well on all metrics. Reducing the PCA of the parameters resulted in a decrease in performance for precision and recall rate, while the false positive rate increased slightly (see Table 3).

[0135] Table 3. Results of the classifier model on the validation dataset. Precision = True positives / (True positives + False positives; or what percentage of frames identified as licking actually showed licking?). Recall rate = True positives / (True positives + False negatives; or what percentage of frames found to actually show licking?). False positive rate = the proportion of "no-licking" frames incorrectly identified as licking.

[0136]

[0137]

[0138] To determine whether 12 points for body parts were necessary for optimal performance, the GentleBoost model was retrained with inputs computed from either 8 points (with points removed from both forepaws and the inner points from both hindaws) or 5 points (with points also removed from the mouth and the outer points from both hindaws). Reducing the number of points resulted in a slight performance penalty, but the 8-point model was very similar to the full 12-point model. PCA analysis of the 8-point and 5-point models showed a significant loss in precision and call rate, with a slight increase in the false positive rate.

[0139] The full GentleBoost classifier using all 12 mouse body parts and all statistical parameters (1047 inputs) achieves the best performance (see Table 3). However, labeling all 12 points to train the tracking module is a time-consuming task, which can be reduced to fewer points if a slight performance loss is an acceptable trade-off. The efficiency gains are insufficient to offset the performance loss of PCA, as evaluating the full 12-point classifier is inexpensive (predicting at approximately 10,000 observations per second). On a laptop computer, opening the Excel file data, calculating the inputs, classifying the behavior, calculating binning, and saving the results in three different formats (HDF5 file, Excel spreadsheet, and a backup of the Matlab output structure file) using Matlab takes approximately 20 to 25 seconds per mouse. A smaller list of parameters is more efficient, but given the low cost of keeping all parameters low, even a small loss in accuracy for detecting licking does not seem to be guaranteed.

[0140] Other classifier models

[0141] Two other classifiers were tested using all 1047 input parameters (see Table 3): a k-nearest neighbor (kNN) classifier (nearest neighbor count = 1, Euclidean distance, equal distance weights, minimum break tie; prediction speed of 110 observations per second); and an ensemble subspace kNN classifier (30 learners, subspace dimension = 624, prediction speed of 8.7 observations per second). Both models performed almost as well as the full GentleBoost model, but were less efficient in implementing predictions. A support vector machine (SVM) with cubic kernels was more efficient than the kNN model (1600 observations per second), but with slightly lower accuracy.

[0142] Model parameters

[0143] The 12-point GentleBoost model has 1047 inputs, but only 385 of them actually contribute information to the classifier. Each useful input contributes only a small amount of information and has no explicit cues. Each of the 12 body parts and 3 time windows is included multiple times in these 1047 inputs, and... Figure 10 The heatmap shows the percentage of useful representations as a proportion of all possible opportunities for that variable. The time window with the most information is a 21-frame (700ms) window, which includes approximately 46% of all cues used. Licking behavior typically lasts for more than a second, so a 700ms window seems sufficient to capture ongoing behavior. Figure 10 The heatmaps show the relative importance of each window size and each body part to the model, with the body part and window size heatmaps highlighting the actual contribution of each to the classifier's decision as a percentage of each's possible contribution.

[0144] Examining the relative information content of body part points can be used to determine the most valuable points to retain for this type of model (see [link]). Figure 10 All the points on the right hind paw, mouth, and nose appear to provide a high level of useful information about the contact between the right hind paw and mouth. Other body points contribute information about body shape, and while the bottom point on the left hind paw can be used for relative comparison (average usage 36%), it may not be necessary to include all three points on the left hind paw (where no formalin was applied), as the outer point (average usage 30%) and inner point (average usage 16%) are used the least. However, including the forepaws seems more useful, as mice frequently grasp their hind paws with their forepaws while licking.

[0145] Short video classifier verification

[0146] The GentleBoost classifier was then tested on 111 new short video clips (from 111 different mice: including 71 brand-new videos and new clips from 40 training videos) for a total testing time of approximately 284 minutes. Licking behavior in each video was manually annotated at a one-second temporal resolution for a single mouse in the activity area. Mice were from all enclosures, and approximately the same number of activity areas were annotated (see [link to documentation]). Figure 9 ). Figure 11A and Figure 11B Results for two videos are shown, directly comparing manual / human-based classification of licking behavior in the videos with classification by an automated phenotypic analysis system. The temporal resolution of manual / human-based classification is worse than that of the model; therefore, if the automated phenotypic analysis system matches the manual classification within + / - 15 frames (i.e., within one second), the licking behavior is recorded as a match. Figure 12 The figure shows the percentage of frames that are consistent between the automated phenotyping system and manual classification for all 111 videos.

[0147] The 43 video clips did not show licking behavior, and the average consistency of these videos was 98.8%, indicating a low level of false positives. Figure 12 The matching accuracy between the two videos was low, with performance within the 84% agreement range. Careful examination of these videos revealed ambiguous behaviors, making manual classification difficult to determine whether the mouse was licking. For example, in one video, the mouth was clearly in contact with the right hind paw, but this was obscured by the tail, so licking could only be inferred rather than categorized by human observation. Another video showed multiple grooming movements in the leg and paw areas, but it was difficult to score this as purely paw-licking behavior. These behaviors were not typical, but difficult to classify, and in these cases, different human observers disagreed with each other.

[0148] Between-observer verification

[0149] To test the reliability among human observers, two observers annotated 60 minutes of video recordings of three mice, visualized using Noldus Media Recorder 4 software. Observations were summed every 5 minutes, and the correlation between observers was generally good (Pearson r = 1.0, 0.82, and 0.97). The two human observers agreed on the constituent actions of licking, but disagreed on the exact timing of starting and stopping the recording / labeling of the behavior. Therefore, licking episodes were sometimes rated as continuous actions by one observer, but as a series of brief episodes by the other. These differences in observer ratings led to several significantly different measurement results in mice 9: for example, observer 1 recorded two 5-minute bins as 67 seconds and 60 seconds, while observer 2 recorded 197 seconds and 151 seconds (see [link to relevant documentation]). Figure 13 Comparing the classifications of the two human observers with those of the automated phenotyping system again revealed a high degree of consistency (Observer 1 Pearson r = 0.98, 0.75, 0.95; Observer 2 Pearson r = 0.98, 0.96, 0.99). For mice 9, the automated phenotyping system appeared to have better consistency with Observer 2, with the aforementioned 5-minute binning recorded as 213 seconds and 141 seconds of licking (see [link to relevant documentation]). Figure 13 ).

[0150] Strain Comparison Verification

[0151] Manual classification was used in the formalin test to compare the licking responses of C57BL / 6NCrl and C57BL / 6J mice in phases I and II. Male C57BL / 6NCrl mice showed a reduced licking response in phase II of the nociceptive response (measured from 20 to 45 minutes), but no significant difference was found in females. To validate the effectiveness of the automated phenotypic analysis system under experimental conditions, a formalin test was performed, comparing similar mouse strains (Jackson Laboratories, USA: C57BL / 6NJ males = 45, females = 30; C57BL / 6J males = 46, females = 30). Because the mice in this study were injected after anesthesia, the first five minutes of the nociceptive response (referred to as phase I) were atypical and were not included in the analysis. All data is processed through the system, but the automated phenotyping system determines the starting frame for each mouse, then skips 9,000 frames (approximately 5 minutes), and then bins the data into 17 five-minute bins (5-10:85-90) of cumulative licking behavior in seconds.

[0152] Figure 14 and Figures 15A to 15DThe data graph presents a comparison of licking behavior between male and female mice in strains C57BL / 6J and C57BL / 6NJ. Figure 14 The summation of licking behavior in a single box during the period from 20 to 45 minutes post-injection is shown; mean and SEM are shown separately by sex (significant two-way ANOVA sex x strain interaction 0.05; * indicates significant post-hoc comparison between each sex between strains). Figure 15A The mean and SEM of licking behavior in females of both strains at 90 minutes post-injection in a 5-minute box (box range from 5-10 minutes to 85-90 minutes) are shown. Figure 15B The mean and SEM of licking behavior in males of both strains at 90 minutes post-injection in 5-minute chambers (chamber range from 5–10 minutes to 85–90 minutes) are shown. Figure 15C The percentage of bootstraps from the significance t-test (α = 0.05) is shown, representing the measure of binning over a time period of 20 to 45 minutes, and the differences between strains due to sample size. Figure 15D The percentage of bootstraps from the significance t-test (α = 0.05) is shown, representing the measure of binning over a time period of 10 to 60 minutes, and the differences between strains due to sample size.

[0153] Figure 14 Summed licking behavior was shown in a time chamber (20 to 45 minutes post-injection), where male C57BL / 6NJ mice showed reduced licking behavior compared to male C57BL / 6J mice. However, female mice showed the opposite pattern, with C57BL / 6NJ mice licking more (sex x strain interaction F). (1,147) =9.99p=0.0019; Holms-Sidak multiple comparisons for males and females p=0.042). The time course of the response over the entire 90 minutes reveals the differences between sex and strain more clearly (see Figure 15A and Figure 15B These curves differ in the timing and magnitude of licking, and how the data is binned for analysis will determine whether to detect differences between sexes or strains. Figure 15C and Figure 15D Bootstrap statistics were compared for two different box selections as sample size increased (α level 0.05). Replicating the previously described box durations of 20 to 45 minutes showed that the probability of finding significant differences between strains increased with sample size, for both males and females (see [link to documentation]). Figure 15CFor females, this choice of box size appears to maximize the differences in response onset timing between strains. Larger box sizes of 10 to 60 minimize the timing differences among females, and bootstrap comparisons show that increasing the sample size does not change the statistical significance shown. Figure 15D The probability of females remaining around 5% is likely due to chance (α = 0.05); females exhibit the same amount of licking. However, the probability of males detecting differences increases with sample size, and males appear to differ in both the amount of licking and the amplitude and duration of peak behavior.

[0154] One strategy is to use a single summing box to examine the Phase II period. While this strategy may be the best option for revealing general differences in the duration of settling and licking, it risks missing information about differences in behavioral timing across different periods. The choice of box duration and the start time of the Phase II analysis can vary, for example, 10–30, 10–60, 10–90, 10–45, 15–45, 20–45, or 20–60. Bootstrapping with different boxes suggests that box selection can lead to inconsistent results if there are temporal differences between strains, sexes, or treatments of interest. The mice in this experiment were anesthetized, which may also have affected the timing of the behavior, as the early portion of the response was significantly reduced. Automated phenotypic analysis showed that male C57BL / 6N licked less, independent of box selection, but box size significantly affected the results for females. Experiments using C57BL / 6N or C57BL / 6N mice as controls need to consider sex separately, as they exhibit significant timing differences in Phase II. Automated phenotyping systems allow experimenters to easily extend the duration of formalin experiments without incurring the costs of annotating long videos. Given the potential for anesthetic effects and the informed choice regarding chamber size, the time difference can be significant over longer durations (60 or 90 minutes).

[0155] Example 2: Devices and Systems

[0156] An automated phenotyping system may employ one or more trained machine learning models, which can take many forms, including neural networks. A neural network may comprise multiple layers, from input to output. Each layer is configured to take a specific type of data as input and output another type of data. The output from one layer is used as the input to the next layer. While the values ​​of the input / output data for a particular layer are not known until the neural network is actually run during runtime, the data describing the neural network describes the structure, parameters, and operations of its multiple layers.

[0157] One or more intermediate layers in a neural network can also be called hidden layers. Each node in a hidden layer is connected to every node in the input layer and every node in the output layer. In the case where the neural network includes multiple intermediate networks, each node in a hidden layer will be connected to every node in the next higher layer and the next lower layer. Each node in the input layer represents a potential input to the neural network, and each node in the output layer represents a potential output of the neural network. Each connection from one node to another node in the next layer can be associated with a weight or score. A neural network can output a single output or a weighted set of possible outputs.

[0158] In one aspect, neural networks can be constructed using recursive connections, such that the outputs of the hidden layers are fed back into the hidden layers for the next set of inputs. Each node in the input layer is connected to every node in the hidden layers. Each node in the hidden layers is connected to every node in the output layer. The outputs of the hidden layers are fed back into the hidden layers for the processing of the next set of inputs. Neural networks containing recursive connections can be called recurrent neural networks (RNNs).

[0159] In some implementations, the neural network may be a Long Short-Term Memory (LSTM) network. In some implementations, the LSTM may be a bidirectional LSTM. A bidirectional LSTM operates on inputs from two time directions, one from past state to future state and the other from future state to past state, where the past state may correspond to the characteristics of the video data in a first time frame, and the future state may correspond to the characteristics of the video data in a second subsequent time frame.

[0160] The processing performed by a neural network is determined by the learned weights on the input of each node and the network structure. Given a specific input, the neural network determines the output of one layer at a time until the output of the entire network layer is computed.

[0161] Connection weights can initially be learned by the neural network during training, where a given input is associated with a known output. In a set of training data, multiple training instances are fed into the network. Each instance typically has its weights set to 1 for the correct connections from input to output, and all other connections given weights of 0. As instances from the training data are processed by the neural network, the input can be sent to the network and compared with its associated output to determine how the network's performance differs from the target performance. Using training techniques, such as backpropagation, the weights of the neural network can be updated to reduce the errors the network introduces when processing the training data.

[0162] Various machine learning techniques can be used to train and operate models to perform the steps described in this paper, such as user identification feature extraction, encoding, user identification scoring, and user identification confidence determination. Models can be trained and operated using a variety of machine learning techniques. Such techniques can include, for example, neural networks (such as deep neural networks and / or recurrent neural networks), inference engines, trained classifiers, etc. Examples of trained classifiers include Support Vector Machines (SVMs), neural networks, decision trees, combinations of AdaBoost ("Adaptive Boosting") and decision trees, and random forests. Taking SVM as an example, SVM is a supervised learning model with an association learning algorithm that analyzes data and identifies patterns in the data; it is commonly used for classification and regression analysis. Given a set of training instances, each labeled as belonging to one of two categories, the model built by the SVM training algorithm assigns new instances to one category or the other, making it a non-probabilistic binary linear classifier. More complex SVM models can be built using training sets labeled with more than two categories, where the SVM determines which category is most similar to the input data. An SVM model can be mapped to separate instances into distinct classes using clear gaps. New instances are then mapped into that same space, and their class is predicted based on which side of the gap they fall on. The classifier can emit a "score" indicating the class that the data most closely matches. This score provides an indication of how well the data matches the class.

[0163] In order to apply machine learning techniques, the machine learning process itself needs to be trained. Training the machine learning components (such as, in this case, one of the first or second models) requires establishing a "benchmark truth" for the training instances. In machine learning, the term "benchmark truth" refers to the accuracy of the classification of the training set by a supervised learning technique. Various techniques can be used to train models, including backpropagation, statistical learning, supervised learning, semi-supervised learning, randomized learning, or other known techniques.

[0164] Figure 16 It is a conceptual block diagram showing a device 1600 that can be used with the system. Figure 17This is a block diagram conceptually illustrating example components of a remote device (such as System 150) that can assist in processing video data, identifying subject behavior, etc. System 150 may include one or more servers. As used herein, "server" may refer to a conventional server as understood in a server / client computing architecture, but may also refer to multiple different computing components that can assist the operations discussed herein. For example, a server may include one or more physical computing components (such as rack servers) that are physically and / or connected to other devices / components via a network and are capable of performing computational operations. A server may also include one or more virtual machines that emulate a computer system and run on one or more devices. A server may also include other combinations of hardware, software, firmware, etc., to perform the operations discussed herein. A server may be configured to operate using one or more of the following computing technologies: client-server model, computer zone model, grid computing, fog computing, mainframe technology, utility computing, peer-to-peer model, sandboxing, or other computing technologies.

[0165] The overall system disclosed herein may include multiple systems 150, such as one or more systems 150 for performing point / body part tracking, one or more systems 150 for frame-level feature extraction, one or more systems 150 for behavior classification, one or more systems 150 for training / configuring an automated phenotypic analysis system, etc. In operation, each of these systems may include computer-readable and computer-executable instructions residing on the respective device 150, as will be further discussed below.

[0166] Each of these devices (1600 / 150) may include one or more controllers / processors (1604 / 1704), each controller / processor may include a central processing unit (CPU) for processing data and computer-readable instructions and a memory (1606 / 1706) for storing data and instructions for the corresponding device. The memory (1606 / 1706) may each include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (1600 / 150) may also include a data storage unit (1608 / 1708) for storing data and controller / processor executable instructions. Each data storage unit (1608 / 1708) may each include one or more non-volatile memory types, such as magnetic storage, optical storage, solid-state storage, etc. Each device (1600 / 150) can also be connected to removable or external non-volatile memory and / or storage devices (such as removable memory cards, memory key drivers, networked storage devices, etc.) via the corresponding input / output device interface (1602 / 1702).

[0167] The computer instructions for operating each device (1600 / 150) and its various components can be executed by the controller / processor (1604 / 1704) of the corresponding device during runtime using memory (1606 / 1706) as a temporary "working" storage device. The computer instructions for the device can be stored in a non-transitory manner in non-volatile memory (1606 / 1706), storage devices (1608 / 1708), or external devices. Alternatively, in addition to or instead of software, some or all of the executable instructions can be embedded in the hardware or firmware of the corresponding device.

[0168] Each device (1600 / 150) includes an input / output device interface (1602 / 1702). Various components can be connected via the input / output device interface (1602 / 1702), as will be discussed further below. Additionally, each device (1600 / 150) may include an address / data bus (1624 / 1724) for transferring data between components within the corresponding device. Besides connecting to other components via the bus (1624 / 1724) (or alternatively), each component within a device (1600 / 150) can also be directly connected to other components.

[0169] See Figure 16 Device 1600 may include an input / output device interface 1602 that connects to various components, such as audio output components, such as a speaker 1612, wired or wireless headphones (not shown), or other components capable of outputting audio. Device 1600 may additionally include a display 1616 for displaying content. Device 1600 may also include a camera 1618.

[0170] The input / output device interface 1602 can connect to one or more networks 199 via antenna 1614, through a wireless local area network (WLAN) radio (such as WiFi), Bluetooth, and / or a wireless network radio (such as a radio capable of communicating with wireless communication networks, such as LTE networks, WiMAX networks, 3G networks, 4G networks, 5G networks, etc.). Wired connections such as Ethernet can also be supported. Through network 199, the system can be distributed in a networked environment. The I / O device interfaces (1602 / 1702) may also include communication components that allow data exchange between devices (such as different physical servers or other components in a server set).

[0171] Components of device 1600 or system 150 may include their own dedicated processors, memory, and / or storage devices. Alternatively, one or more components of device 1600 or system 150 may utilize the I / O interface (1602 / 1702), processor (1604 / 1704), memory (1606 / 1706), and / or storage device (1608 / 1708) of device 1600 or system 150, respectively.

[0172] As noted above, multiple devices can be employed in a single system. In such a multi-device system, each device may include different components for performing different aspects of system processing. Multiple devices may include overlapping components. The components of device 1600 and system 150 described herein are illustrative and may be positioned as independent devices or may be included, wholly or partially, as components of a larger device or system.

[0173] The concepts disclosed herein can be applied to many different devices and computer systems, including, for example, general-purpose computing systems, video / image processing systems, and distributed computing environments.

[0174] The foregoing aspects of this disclosure are intended to be illustrative. They have been chosen to explain the principles and applications of this disclosure and are not intended to be exhaustive or limiting. Many modifications and variations to the aspects disclosed herein will be apparent to those skilled in the art. Those of ordinary skill in the art of computer and speech processing will recognize that the components and process steps described herein can be interchanged with other components or steps, or combinations of components or steps, and the benefits and advantages of this disclosure will still be achieved. Furthermore, it will be apparent to those skilled in the art that this disclosure can be practiced without some or all of the specific details and steps disclosed herein.

[0175] Aspects of the system disclosed in this invention can be implemented as a computer method, or as an article of manufacture such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be computer-readable and may include instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage medium may be implemented as volatile computer memory, non-volatile computer memory, hard disk drive, solid-state memory, flash memory drive, removable disk, and / or other media. Furthermore, components of the system may be implemented in firmware or hardware.

[0176] Example 3. Automated measurement of licking; comprehensive nociceptivity index; testing of genetic variations in nociceptivity response.

[0177] The licking action was automatically measured in widely used open field activity spaces with top-down camera views. Furthermore, multiple potential harm-prevention behaviors were automatically measured to obtain a comprehensive harm sensitivity index.

[0178] method

[0179] The method used is as described in Example 1, except for steps otherwise described below.

[0180] Open field video data acquisition, frame-by-frame measurement, and feature analysis.

[0181] As described above in Example 1 and as previously stated ( Figure 18A [Kumar, V. et al., PNAS 108, 15557–15564. ISSN: 0027-8424 (2011); Geuther, B. et al., Communications Biology 2, 124 (March 2019)], top-down video data of each mouse during an hour of open field activity were collected 104. The open field videos were processed by a pose estimation network and a tracking network based on a deep neural network to generate a 12-point pose skeleton and an elliptical fitted trajectory for each frame of the mouse [Sheppard, K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020); Geuther, B. et al., Communications Biology 2, 124 (March 2019)]. These per-frame measurements were used to create a behavior classifier and to engineer features such as anxiety, hyperactivity, traditional open field measurements [Geuther, B. et al., Communications Biology 2, 124 (March 2019)], neural network-based grooming [Geuther, BQ et al., Elife 10, e63207 (2021)], and novel gait measurements [Sheppard, K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Figure 22 Information on video features used in certain embodiments of the invention is provided.

[0182] result

[0183] Genetic variation in nociceptive responses is also of interest. Changes in licking behavior in inbred mouse strains in response to formalin have been previously described [Mogil, J. et al., Pain 80, 67–82 (1999)]. Strains were selected from a range of high-level licking responders (C57BL6J and C3HHeJ) to low-level licking responders (BALBcJ and AJ) [Mogil, J. et al., Pain 80, 67–82 (1999)]. Both male and female mice from each strain were used, as sex differences in response have also been described. A dataset of 194 mice was obtained by testing at least five male and five female mice from each strain at four formalin doses (0.00%, 1.25%, 2.50%, and 5.00%) for each dose. Figure 18A ).

[0184] One behavior classified using JABS is licking / biting. Licking is considered the most important anti-harm behavior for quantification in formalin assays [Saddi, G.-M. and Abbott, F., Pain 89, 53–63 (2001); Abbot, F. et al., Pain, 83, 561–569 (1999); Wotton, JM et al., Molecular Pain 16, 1744806920958596 (2020)]. The licking classifier was trained on a set of videos from a previous experiment using male and female C57BL6 / J mice and four formalin doses of 0.00% (saline), 0.27%, 0.87%, and 2.5%. A behavioral scientist intensively labeled four one-hour videos of each of the four strains after administration of a 5.00% formalin dose, for a total of four hours of video. A classifier was used on these four videos, and it was found that all the videos had a high degree of frame consistency. Figure 18B The consistency at the frame level is shown. To further investigate the consistency between the labeler and the classifier, the overlap between licking episodes detected by the classifier and the labeler was compared. For each episode detected by the classifier or the labeler, the number of frames in which the classifier and the labeler agree that licking is occurring is calculated. If at least half of the frames are consistent, the episode is considered to be overlapping. Figure 18C The percentage of overlapping licking episodes lasting longer than 1 second is shown. Using frame-level classification and labeling, the number of licking episodes, the time spent licking, and the average duration of each licking episode are calculated for each video. Figures 18D to 18F These measurements were compared between the classifier and the labeled data set, showing that while the classifier was slightly more conservative than the labeled data set for most videos, the amount of licking detected was generally comparable. Taken together, these results suggest that this is a reasonable licking classifier.

[0185] Next, the licking measurements for each video in the dataset were categorized and calculated. When examining the differences in licking time between male and female mice under different dosages and strains, significant differences were observed between different strains and sexes. Figures 18G to 18H As previously mentioned, C57BL6 / J mice exhibit high levels of licking [Mogil, J. et al., Pain 80, 67–82 (1999)], C3HHeJ mice exhibit moderate levels of licking, and BALBcJ and AJ mice exhibit low levels of licking. While licking is a good measure of dose-dependent nociception in certain mouse strains, it may be less reliable for low-responders such as BALBcJ and AJ mice. Because a comprehensive nociception score measuring multiple behaviors has been found to be a more robust method for quantifying nociception in formalin assays, various automated measurements were investigated.

[0186] In formalin measurements, trembling paws is another known injury prevention behavior. A classifier was trained on trembling behavior. Figure 19A and Figure 19B The time taken by males and females to shake their paws at different doses is shown separately. Next, it is assumed that distressed mice may not stand upright very often, and a classifier is trained for wall-supported upright behavior. Figure 19C and Figure 19D The time taken for males and females to stand upright is shown separately; Figure 19E The accuracy measurements of the 10-fold cross-validation are shown. Mice administered high doses of formalin were found to exhibit stasis episodes, remaining motionless for several seconds at a time. Stasis episodes were heuristically identified by taking the average velocity at points on the nose, base of the head, and tail root in each frame and finding periods where the mouse's average velocity was close to zero for at least three seconds. Several characteristics were calculated based on these measurements. Figure 19F and Figure 19GThe time taken for 3 to 6 seconds of stasis episodes in male and female mice at different doses is shown separately. Interestingly, BALBcJ and AJ mice were more reactive than C57BL6 / J mice in terms of stasis episodes. Gait was also examined by extracting stride measurements from mice moving freely in an open field using previously described methods [Sheppard, K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780(2020)]. For many BALBcJ and AJ mice, stride was rarely or not observed at all throughout the video. Spontaneous activity is known to be low in both strains. Therefore, for BALBcJ and AJ mice, almost all measurements related to movement in an open field were not significantly correlated with formalin dose ( Figure 20 Examining the correlation between all measurements revealed significant strain differences between high- and low-responders. Figure 20 ).

[0187] Next, use Figure 22 The features in the model were fitted to a logit-linked model [Agresti, A. Categorical data analysis (John Wiley & Sons, 2003)] to an ordinal response (dose). Feature weights / coefficients (β) extracted from the model were used to construct a univariate pain scale. Next, the data were projected onto the univariate pain scale axis along with ordinal category labels (dose). Figure 21A A vertical dashed line corresponding to the intercept from the cumulative link model was plotted, separating dose levels 1 and 2. Figure 21A The vertical dashed line is then used to separate animals belonging to the no / low pain (dose level 0, 1) and high pain (dose level 2, 3) groups in the binary classification analysis below. Figure 21C and Figure 21D The contribution of each feature to the univariate pain scale is obtained using the feature coefficients / weights (β). Figure 21B A binary logistic regression model was constructed using different feature sets [“open field” (gray solid dots); “other”, including engineered features and features obtained from the behavior classifier (“X”); and “all”, including the open field and both (“*”)]. Next, the accuracy metric of this classifier, obtained using leave-one-out cross-validation in animals, was used to evaluate the efficacy of the pain scale in classifying animals into low (dose level 0, 1) pain groups and high (dose level 2, 3) pain groups. Figure 21CIt was found that including all features ("*") yielded slightly better accuracy in classifying animals belonging to the low-pain and high-pain groups compared to including only the "other" feature set ("X"). (Using...) Figure 21C A similar procedure, but the classifier is trained on animals belonging to all but one strain (strain-leave-one cross-validation). Compare the classes used to classify animals (belonging to the remaining strains) into their corresponding pain groups. Figure 21D Different feature sets (similar to) Figure 21C The performance of the "All" feature set ("*") was evaluated to assess differences due to strain. As previously mentioned, the "All" feature set ("*") provided a slight improvement over the "Other" feature set in predicting pain levels. In fact, for C57BL / 6NJ, the "Other" feature set outperformed the "All" feature set. Another interesting finding was that the open field measure (gray solid dots) had different predictive accuracies for different strains. For example, classifying C57BL / 6NJ animals into low-pain / high-pain groups using the open field measure achieved an accuracy of up to 60% ( Figure 21D In contrast, open field measurements are unreliable for predicting pain levels in AJ animals.

[0188] Principle of Equivalence

[0189] Although several embodiments of the invention have been described and illustrated herein, those skilled in the art will readily conceive of a variety of other means and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the invention. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using the teachings of this invention. Those skilled in the art will recognize, or can determine, many equivalents of the specific embodiments of the invention described herein using only conventional experimentation. Therefore, it should be understood that the foregoing embodiments are presented by way of example only and within the scope of the appended claims and their equivalents; the invention can be practiced in ways other than those specifically described and claimed. The invention relates to each individual feature, system, article, material, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, and / or methods (if such features, systems, articles, materials, and / or methods are not contradictory) is included within the scope of the invention. All definitions used herein should be understood to have higher priority than dictionary definitions, definitions in references incorporated herein, and / or the general meaning of these defined terms.

[0190] The indefinite articles “a” and “an” as used herein in the specification and claims shall be understood to mean “at least one / a” unless the contrary is explicitly stated. The phrase “and / or” as used herein in the specification and claims shall be understood to mean “any one or both” of the elements so combined, that is, elements that exist together in some cases and separately in others. Optional elements other than those specifically indicated by the “and / or” clause may exist, whether or not they are related to those specifically indicated, unless the contrary is explicitly stated.

[0191] The conditional language used herein (wherein words such as “can,” “may,” “may,” “perhaps,” “for example,” etc.) are generally intended to convey, unless otherwise expressly stated or understood in the context in which they are used, that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are necessary for one or more embodiments in any way, nor is it intended to imply that one or more embodiments must include logic for determining, with or without further input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive sense (rather than its exclusive sense) such that, when used, for example, to connect lists of elements, the term “or” means one, some, or all of the elements in that list.

[0192] All references, patents and patent applications and publications cited or mentioned in this application are incorporated herein by reference in their entirety.

Claims

1. A computer-implemented method, comprising: Receive video data representing the subject's video capture motion; Using the video data, a first point data is determined for the first frame during a first time period, the first point data identifying the location of a first body part of the subject; Using the video data, a second point data is determined for the first frame, the second point data identifying the location of a second body part of the subject; Using the video data, a third point data is determined for the first frame, the third point data identifying the location of a third body part of the subject; First distance data is determined using the first point data and the second point data, where the first distance data represents the distance between the first body part and the second body part. The first point data and the third point data are used to determine the second distance data, which represents the distance between the first body part and the third body part; Determine a first feature vector that corresponds at least to the first frame and the second frame, wherein the first feature vector includes at least the first distance data and the second distance data; The first angle data is determined using the first point data, the second point data, and the third point data. The first angle data represents the angle corresponding to the first body part, the second body part, and the third body part. and Determine a second feature vector that corresponds at least to the first frame, the second feature vector including at least the first angle data; The trained model is used to process at least the first feature vector and the second feature vector, the trained model being configured to identify the likelihood that the subject will exhibit a behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. as well as Based on processing at least the first feature vector and the second feature vector, a first label corresponding to the first time period is determined. The first label identifies the subject's first behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. The first feature vector, which includes distance data, and the second feature vector, which includes angle data, are processed to determine whether the subject exhibits nociceptive or anti-nociceptive behavior based on the position of one or more body parts relative to other body parts.

2. The computer-implemented method according to claim 1 further includes: The video data is used to determine a fourth point of data for the second frame during the first time period, the fourth point of data identifying the position of the first body part; The video data is used to determine a fifth point of data for the second frame, the fifth point of data identifying the position of the second body part; The video data is used to determine a sixth point of data for the second frame, the sixth point of data identifying the position of the third body part; Using the fourth point data and the fifth point data, a third distance data is determined for the second frame, the third distance data representing the distance between the first body part and the second body part; Using the fourth point data and the sixth point data, a fourth distance data is determined for the second frame, the fourth distance data representing the distance between the first body part and the third body part; Using the fourth point data, the fifth point data, and the sixth point data, second angle data is determined for the second frame, the second angle data representing the angles corresponding to the first body part, the second body part, and the third body part; and The second feature vector is determined to include at least the third distance data, the fourth distance data, and the second angle data.

3. The computer-implemented method according to claim 1, wherein the second distance data represents the distance between the first body part and the second body part for the second frame during the first time period.

4. The computer-implemented method according to claim 3 further includes: At least the first distance data and the second distance data are used to calculate the metric data corresponding to the first frame. The first feature vector includes the metric data.

5. The computer-implemented method according to claim 4, wherein the metric data represents at least a statistical analysis corresponding to the first distance data and the second distance data, and the statistical analysis is at least one of the mean, standard deviation, median, and median absolute deviation.

6. The computer-implemented method according to claim 1, further comprising: An additional trained model is used to process the video data to determine the first point data, wherein the first point data includes pixel data representing the location of the first body part.

7. The computer-implemented method according to claim 1, further comprising: The video data is processed using an additional trained model to determine the probability that pixel coordinates correspond to the first body part. as well as The first point data, including the pixel coordinates, is determined at least in part based on the probability of satisfying a threshold.

8. The computer-implemented method according to claim 1, further comprising: Using the video data, additional point data is determined for the first frame, the additional point data identifying the locations of at least 12 parts of the subject, wherein the 12 parts include at least the first body part and the second body part.

9. The computer-implemented method according to claim 8, further comprising: Additional distance data is determined for the first frame, representing distances between multiple pairs of body parts formed using paired portions of the subject's 12 body parts. The first feature vector includes the additional distance data.

10. The computer-implemented method according to claim 8, further comprising: Additional angle data is determined for the first frame. This additional angle data represents angles corresponding to multiple body part triplets, which are formed by selecting three parts from the subject's 12 body parts. The first feature vector includes the additional angle data.

11. The computer-implemented method according to claim 1, further comprising: Determine additional feature vectors corresponding to six frames during the first time period, the six frames including at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

12. The computer-implemented method according to claim 11, further comprising: For the first frame, positional data representing the pixel coordinates of 12 parts of the subject is determined, the positional data including at least the first point data, the second point data, and the third point data, and Using the trained model to process the metric data also includes using the trained model to process the location data.

13. The computer-implemented method according to claim 1, further comprising: Determine additional feature vectors corresponding to 11 frames during the first time period, wherein the 11 frames include at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

14. The computer-implemented method of claim 13, wherein the 11 frames include five frames preceding the first frame and five frames following the first frame.

15. The computer-implemented method according to claim 1, further comprising: Determine additional feature vectors corresponding to 21 frames during the first time period, wherein the 21 frames include at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

16. The computer-implemented method of claim 15, wherein the 21 frames include 11 frames preceding the first frame and 11 frames following the first frame.

17. The computer-implemented method of claim 1, wherein the video data represents video capture motion of more than one subject.

18. The computer-implemented method of claim 1, wherein the trained model is a classifier configured to process feature data corresponding to video frames to determine the behavior exhibited by the subject as presented in the video frames, the feature data corresponding to multiple parts of the subject.

19. The computer-implemented method according to claim 1, wherein: The first body part is the subject's mouth; The second body part is the subject's right hind foot; The trained model is configured to identify the likelihood that the subject will exhibit contact between the first body part and the second body part; and The first label indicates that the first frame represents contact between the first body part and the second body part.

20. The computer-implemented method of claim 1, wherein the first frame corresponds to 30 milliseconds of video data.

21. The computer-implemented method of claim 1, wherein the video data corresponds to a first video capturing a top view of the subject and a second video capturing a side view of the subject.

22. The computer-implemented method of claim 1, wherein the subject is a mammal.

23. The computer-implemented method of claim 1, wherein the subject is a rodent.

24. The computer-implemented method of claim 1, wherein the subject is a primate.

25. A method for determining nociceptive behavior in a test subject, the method comprising monitoring the test subject’s responses, wherein the monitoring comprises the computer-implemented method of claim 1.

26. The method of claim 25, wherein the test subject suffers from pain.

27. The method of claim 26, wherein the pain condition includes one or more of the following: inflammatory pain, neuropathic pain, muscle pain, joint pain, chronic pain, visceral pain, cancer pain, and postoperative pain.

28. The method of claim 25, wherein the test subject is an animal model exhibiting pain symptoms.

29. The method of claim 25, wherein pain is induced in the test subject.

30. The method of claim 29, wherein inducing the pain in the test subject includes inducing inflammation in the test subject.

31. The method of claim 30, wherein inducing inflammation comprises exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, and chemical agents.

32. The method of claim 31, wherein the chemical agent comprises one or more of formalin and acetone.

33. The method of claim 31 or 32, wherein the means of inducing the pain include one or more of the following: exposing the test subject to the chemical agent, and injecting the test subject with the chemical agent.

34. The method of claim 25, wherein the test subject is a genetically engineered test subject.

35. The method of claim 25, wherein the test subject is a rodent.

36. The method of claim 25, wherein the test subject is a mouse.

37. The method of claim 36, wherein the mouse is a genetically engineered mouse.

38. The method of claim 25, further comprising administering the candidate therapeutic agent to the test subject.

39. The method of claim 38, wherein if pain is induced in the test subject, the candidate therapeutic agent is administered to the test subject before the pain is induced in the test subject.

40. The method of claim 38, wherein if pain is induced in the test subject, the candidate therapeutic agent is administered to the test subject after the pain is induced in the test subject.

41. The method of claim 25, wherein the monitoring results of the test subject are compared with control results.

42. The method of claim 41, wherein the control results are from control subjects monitored using the computer-implemented method.

43. The method of claim 42, wherein pain is induced in the control subject.

44. The method of claim 42, wherein the control subject is an animal model exhibiting pain symptoms.

45. The method of claim 43 or 44, wherein the control subject was not given the candidate therapeutic agent.

46. ​​The method of claim 43, wherein the dose of the candidate therapeutic agent administered to the control subject is different from the dose of the candidate therapeutic agent administered to the test subject.

47. The method of claim 43, wherein the control result is the result of prior monitoring of the test subject using the computer-implemented method.

48. The method of claim 25, wherein the monitoring of the subject identifies the subject's chronic pain condition.

49. The method of claim 25, wherein the monitoring of the subject identifies the efficacy of the candidate therapeutic agent in treating pain symptoms.

50. A non-therapeutic method for identifying the efficacy of a candidate therapeutic agent in treating a subject's pain condition, comprising: The candidate therapeutic agent was administered to the test subjects, and The test subjects are monitored, wherein the test subjects are animal models of pain conditions or genetically engineered test subjects, wherein the monitoring includes the computer-implemented method of claim 1, and wherein the monitoring results indicating pain relief in the test subjects identify the efficacy of the candidate therapeutic agent in treating the pain conditions.

51. The method of claim 50, wherein the pain condition includes one or more of the following: inflammatory pain, neuropathic pain, muscle pain, joint pain, chronic pain, visceral pain, cancer pain, and postoperative pain.

52. The method of claim 50, wherein the test subject is a rodent.

53. The method of claim 50, wherein the test subject is a mouse.

54. The method of claim 50, wherein the test subject is a genetically engineered mouse.

55. The method of claim 50, wherein pain is induced in the test subject prior to the monitoring.

56. The method of claim 50, wherein inducing the pain in the test subject includes inducing inflammation in the test subject.

57. The method of claim 56, wherein inducing inflammation comprises exposing the test subject to one or more of the following stimuli: heat, light, pressure, cold, and chemical agents.

58. The method of claim 57, wherein the chemical agent comprises one or more of formalin and acetone, and wherein the means of inducing the pain comprises one or more of: exposing the test subject to the chemical agent, and injecting the test subject with the chemical agent.

59. The method of claim 55, wherein the candidate therapeutic agent is administered to the test subject prior to inducing the pain in the test subject.

60. The method of claim 55, wherein the candidate therapeutic agent is administered to the test subject after the pain is induced in the test subject.

61. The method of claim 50, wherein the monitoring results of the test subject are compared with control results.

62. The method of claim 61, wherein the control results are from control subjects monitored using the computer-implemented method.

63. The method of claim 62, wherein pain is induced in the control subject.

64. The method of claim 62, wherein the control subject has the pain condition.

65. The method of claim 64, wherein the control subject is an animal model of the pain condition.

66. The method of claim 62, wherein the control subject was not administered the candidate therapeutic agent.

67. The method of claim 62, wherein the dose of the candidate therapeutic agent administered to the control subject is different from the dose of the candidate therapeutic agent administered to the test subject.

68. The method of claim 62, wherein the control result is the result of prior monitoring of the test subject using the computer-implemented method.

69. The method of claim 50, further comprising further testing the efficacy of the candidate therapeutic agent.

70. A system comprising: At least one processor; as well as At least one memory includes the following instructions, which, when executed by the at least one processor, cause the system to: Receive video data representing the subject's video capture motion; Using the video data, a first point data is determined for the first frame during a first time period, the first point data identifying the location of a first body part of the subject; Using the video data, a second point data is determined for the first frame, the second point data identifying the location of a second body part of the subject; Using the video data, a third point data is determined for the first frame, the third point data identifying the location of a third body part of the subject; First distance data is determined using the first point data and the second point data, where the first distance data represents the distance between the first body part and the second body part. The first point data and the third point data are used to determine the second distance data, which represents the distance between the first body part and the third body part; Determine a first feature vector that corresponds at least to the first frame and the second frame, wherein the first feature vector includes at least the first distance data and the second distance data; The first angle data is determined using the first point data, the second point data, and the third point data. The first angle data represents the angle corresponding to the first body part, the second body part, and the third body part. Determine a second feature vector that corresponds at least to the first frame, the second feature vector including at least the first angle data; The trained model is used to process at least the first feature vector and the second feature vector, the trained model being configured to identify the likelihood that the subject will exhibit a behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. as well as Based on processing at least the first feature vector and the second feature vector, a first label corresponding to the first time period is determined. The first label identifies the subject's first behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. The first feature vector, which includes distance data, and the second feature vector, which includes angle data, are processed to determine whether the subject exhibits nociceptive or anti-nociceptive behavior based on the position of one or more body parts relative to other body parts.

71. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: The video data is used to determine a fourth point of data for the second frame during the first time period, the fourth point of data identifying the position of the first body part; The video data is used to determine a fifth point of data for the second frame, the fifth point of data identifying the position of the second body part; The video data is used to determine a sixth point of data for the second frame, the sixth point of data identifying the position of the third body part; Using the fourth point data and the fifth point data, a third distance data is determined for the second frame, the third distance data representing the distance between the first body part and the second body part; Using the fourth point data and the sixth point data, a fourth distance data is determined for the second frame, the fourth distance data representing the distance between the first body part and the third body part; Using the fourth point data, the fifth point data, and the sixth point data, second angle data is determined for the second frame, the second angle data representing the angles corresponding to the first body part, the second body part, and the third body part; and The second feature vector is determined to include at least the third distance data, the fourth distance data, and the second angle data.

72. The system of claim 70, wherein the second distance data represents the distance between the first body part and the second body part for the second frame during the first time period.

73. The system of claim 72, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: At least the first distance data and the second distance data are used to calculate the metric data corresponding to the first frame. The first feature vector includes the metric data.

74. The system of claim 73, wherein the metric data represents a statistical analysis corresponding at least to the first distance data and the second distance data, the statistical analysis being at least one of the mean, standard deviation, median, and median absolute deviation.

75. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: An additional trained model is used to process the video data to determine the first point data, wherein the first point data includes pixel data representing the location of the first body part.

76. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: The video data is processed using an additional trained model to determine the probability that pixel coordinates correspond to the first body part; and The first point data, including the pixel coordinates, is determined at least in part based on the probability of satisfying a threshold.

77. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Using the video data, additional point data is determined for the first frame, the additional point data identifying the locations of at least 12 parts of the subject, wherein the 12 parts include at least the first body part and the second body part.

78. The system of claim 77, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Additional distance data is determined for the first frame, representing distances between multiple pairs of body parts formed using paired portions of the subject's 12 body parts. The first feature vector includes the additional distance data.

79. The system of claim 77, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Additional angle data is determined for the first frame. This additional angle data represents angles corresponding to multiple body part triplets, which are formed by selecting three parts from the subject's 12 body parts. The first feature vector includes the additional angle data.

80. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Determine additional feature vectors corresponding to six frames during the first time period, the six frames including at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

81. The system of claim 80, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: For the first frame, positional data representing the pixel coordinates of 12 parts of the subject is determined, the positional data including at least the first point data, the second point data, and the third point data, and The instruction that causes the system to use the trained model to process the metric data further causes the system to use the trained model to process the location data.

82. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Determine additional feature vectors corresponding to 11 frames during the first time period, wherein the 11 frames include at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

83. The system of claim 82, wherein the 11 frames include five frames preceding the first frame and five frames following the first frame.

84. The system of claim 70, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Determine additional feature vectors corresponding to 21 frames during the first time period, wherein the 21 frames include at least the first frame and the second frame; The additional feature vector is used to compute metric data, which represents at least one of the mean, standard deviation, median, and median absolute deviation; as well as The trained model is used to process the metric data to determine the first label.

85. The system of claim 84, wherein the 21 frames include 11 frames preceding the first frame and 11 frames following the first frame.

86. The system of claim 70, wherein the video data represents video capture motion of more than one subject.

87. The system of claim 70, wherein the trained model is a classifier configured to process feature data corresponding to video frames to determine the behavior exhibited by the subject as presented in the video frames, the feature data corresponding to multiple parts of the subject.

88. The system according to claim 70, wherein: The first body part is the subject's mouth; The second body part is the subject's right hind foot; The trained model is configured to identify the likelihood that the subject will exhibit contact between the first body part and the second body part; and The first label indicates that the first frame represents contact between the first body part and the second body part.

89. The system of claim 70, wherein the first frame corresponds to 30 milliseconds of video data.

90. The system of claim 70, wherein the video data corresponds to a first video capturing a top view of the subject and a second video capturing a side view of the subject.

91. The system of claim 70, wherein the subject is a mammal.

92. The system of claim 70, wherein the subject is a rodent.

93. The system of claim 70, wherein the subject is a primate.

94. An apparatus for performing a computer-executable method, the method comprising: Receive video data representing the subject's video capture motion; Using the video data, a first point data is determined for the first frame during a first time period, the first point data identifying the location of a first body part of the subject; Using the video data, a second point data is determined for the first frame, the second point data identifying the location of a second body part of the subject; Using the video data, a third point data is determined for the first frame, the third point data identifying the location of a third body part of the subject; First distance data is determined using the first point data and the second point data, where the first distance data represents the distance between the first body part and the second body part. The first point data and the third point data are used to determine the second distance data, which represents the distance between the first body part and the third body part; Determine a first feature vector that corresponds at least to the first frame and the second frame, wherein the first feature vector includes at least the first distance data and the second distance data; The first angle data is determined using the first point data, the second point data, and the third point data. The first angle data represents the angle corresponding to the first body part, the second body part, and the third body part. and Determine a second feature vector that corresponds at least to the first frame, the second feature vector including at least the first angle data; The trained model is used to process at least the first feature vector and the second feature vector, the trained model being configured to identify the likelihood that the subject will exhibit a behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. as well as Based on processing at least the first feature vector and the second feature vector, a first label corresponding to the first time period is determined. The first label identifies the subject's first behavior during the first time period, wherein the behavior is a nociceptive behavior or a nociceptive behavior. The first feature vector, which includes distance data, and the second feature vector, which includes angle data, are processed to determine whether the subject exhibits nociceptive or anti-nociceptive behavior based on the position of one or more body parts relative to other body parts.

Images