Long-term and continuous animal behavioral monitoring

Abstract

To provide systems and methods for continuous monitoring of the behavior of animals, such as small rodents.SOLUTION: A system for animal tracking provides an encoder-decoder segmentation neural network-based tracker which is used with video data obtained by cameras, arena design, environmental sensors, ultrasonic sensors, and monitoring systems. The tracker can be easily extended to other experimental paradigms and even to other animals through transfer learning, thus forming a robust, generalizable solution for bio-behavioral research.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 542,180, entitled "Long-Term and Continuous Animal Behavioral Monitoring," filed August 7, 2017, and U.S. Provisional Patent Application No. 62 / 661,610, entitled "Robust Mouse Tracking In Complex Environments Using Neural Networks," filed April 23, 2018, the entire contents of each of which are incorporated by reference. [Background technology]

[0002]

[0002] Animal behavior can be understood as the output of the nervous system in response to internal or external stimuli. The ability to accurately track animals can be useful as part of the process of classifying animal behavior. For example, behavioral changes are hallmarks of aging, psychiatric disorders, or metabolic diseases, and can reveal important information about the influence of an animal's physiological, neurocognitive, and emotional state. Summary of the Invention

[0003]

[0003] Traditionally, experiments evaluating animal behavior are conducted non-invasively, with researchers directly interacting with the animals. As an example, a researcher may remove an animal, such as a mouse, from its home environment (e.g., a cage) and place it in a different environment (e.g., a maze or other device). The researcher may then observe the animal's task performance by positioning itself near the new environment and tracking the animal. However, it is known that animals may exhibit different behavior in the new environment or different behavior toward the experimenter performing the test. This often leads to confounding data and irreproducible and misleading results.

[0004]

[0004] Minimally invasive monitoring techniques have been developed to minimize human interference during behavioral monitoring experiments. As an example, video monitoring has been investigated for use in monitoring animal behavior. However, challenges remain with video monitoring. On the one hand, a major hurdle remains the ability to continuously capture video data with high spatiotemporal resolution over long periods of time under a wide range of environmental conditions. Animal observation studies conducted over extended periods, such as days, weeks, and / or months, can generate large amounts of data that are costly to acquire and store. On the other hand, even assuming sufficient quality of video data can be acquired and stored, it is economically unfeasible for researchers to manually review the large amount of video footage generated during long-term observations and track animals over such long periods of time. This challenge becomes even more pronounced when the number of animals being observed increases, as may be necessary when screening new drugs or conducting genomics experiments.

[0005]

[0005] To address this problem, computer-based techniques have been developed to analyze captured video of animal behavior. However, existing computer-based systems are unable to accurately track different animals in complex and dynamic environments. As one example, existing computer-based techniques for tracking animals are unable to accurately distinguish one animal from its background (e.g., cage walls and / or floor, objects within the cage such as a water bowl) or to accurately distinguish multiple animals from each other. In the best case scenario, if a given animal is not accurately tracked during an observation period, useful observation data may be lost. In the worst case scenario, if a given animal or part of it is mistracked or mistaken for another during an observation period, valuable observation data may be lost. However, video tracking techniques, such as changing the animal's coat color, can introduce errors into the behaviors classified from the captured video data. Although techniques such as changing the animal's coat color have been employed to facilitate tracking, changing the animal's coat color can also alter the animal's behavior. As a result, existing video tracking methods performed in complex and dynamic environments or with genetically heterogeneous animals require a high level of user involvement, thereby eliminating the aforementioned advantages of video observation. Therefore, large-scale and / or long-term animal monitoring experiments remain unfeasible.

[0006]

[0006] As neuroscience and behavioral science move into an era of large amounts of behavioral data and computational ethology, better techniques for tracking animals are needed to facilitate classification of their behavior over long periods of time and in semi-natural and dynamic environments.

[0007]

[0007] Accordingly, neural network-based systems and methods have been developed that can provide robust and scalable tracking of animals (e.g., mice) in open fields. By way of example, systems and methods are provided that facilitate the acquisition of video data of animal movements with high spatiotemporal resolution. This video data can be captured continuously over long periods of time and under a wide range of environmental conditions.

[0008] The acquired video data can be employed as input for a convolutional neural network architecture for tracking. The neural network can be trained to perform tracking under multiple environmental conditions with high robustness and without user-involved adjustments during training when new environments or animals are presented. Examples of such experimental conditions can include different mouse strains with varying coat colors, body shapes, and behaviors, as well as different cage environments. Thus, embodiments of the present disclosure can facilitate minimally invasive animal tracking, thereby facilitating behavioral monitoring of large numbers of animals over long periods of time under heterogeneous conditions.

[0009] In certain embodiments, the disclosed video observation and animal tracking techniques may be employed in combination, although it may be understood that each of these techniques may be employed alone or in any combination with each other or with other techniques.

[0010] In one embodiment, a method for animal tracking is provided. The method may include receiving, by a processor, video data representing observations of animals; and executing, by the processor, a neural network architecture. The neural network architecture may be configured to receive input video frames extracted from the video data; generate an ellipse description of at least one animal based on the input video frames, the ellipse description being defined by predetermined ellipse parameters; and provide, for the at least one animal, data including values ​​characterizing the predetermined ellipse parameters.

[0011]

[0011] In another embodiment of this method, the ellipse parameters may be coordinates representing the animal's position within a plane, the length of the animal's major axis and the length of its minor axis, and the angle at which the animal's head is pointing, defined relative to the direction of the major axis.

[0012] In another embodiment of this method, the neural network architecture may be an encoder-decoder segmentation network configured to predict a foreground-background segmentation image from an input video frame, predict whether an animal is present in the input video frame based on the segmentation image in a pixel-wise manner, output a segmentation mask based on the pixel-wise prediction, and fit an ellipse to portions of the segmentation mask where an animal is predicted to be present to determine values ​​characterizing predetermined ellipse parameters.

[0013] In another embodiment of this method, the encoder-decoder segmentation network may include a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract an input video frame into a set of small spatial resolution features. The feature decoder may be configured to transform the set of features into a shape consistent with the input video frame and output a foreground-background segmented image. The angle predictor may be configured to predict an angle at which the animal's head is pointing.

[0014]

[0014] In another embodiment of this method, the neural network architecture may comprise a binning classification network configured to predict a heat map of most probable values ​​for each ellipse parameter of the ellipse description.

[0015]

[0015] In another embodiment of this method, the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, which abstraction can be employed to generate a heat map.

[0016]

[0016] In another embodiment of this method, the neural network architecture may comprise a recurrent network configured to extract features from the input video frames and directly predict values ​​characterizing each of the ellipse parameters.

[0017] In another embodiment of this method, the animal can be a rodent.

[0018] In one embodiment, a system for animal tracking is provided. The system may include a data storage device that maintains video data representing observations of animals. The system may also include a processor configured to receive the video data from the data storage device and implement a neural network architecture. The neural network architecture may be configured to receive input video frames extracted from the video data, generate an ellipse description of at least one animal based on the video frames, the ellipse description defined by predetermined ellipse parameters, and provide data for the at least one animal including values ​​characterizing the predetermined ellipse parameters.

[0018]

[0019] In another embodiment of the system, the ellipse parameters may be coordinates representing the animal's position within a plane, the length of the animal's major and minor axes, and the angle at which the animal's head is pointing, defined relative to the direction of the major axis.

[0019]

[0020] In another embodiment of the system, the neural network architecture may be an encoder-decoder segmentation network configured to predict a foreground-background segmentation image from an input video frame, predict whether an animal is present in the input video frame based on the segmentation image in a pixel-wise manner, output a segmentation mask based on the pixel-wise prediction, and fit an ellipse to portions of the segmentation mask where an animal is predicted to be present to determine values ​​characterizing predetermined ellipse parameters.

[0020]

[0021] In another embodiment of this system, the encoder-decoder segmentation network may include a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract an input video frame into a set of small spatial resolution features. The feature decoder may be configured to transform the set of features into a shape consistent with the input video frame and output a foreground-background segmented image. The angle predictor may be configured to predict an angle at which the animal's head is pointing.

[0021]

[0022] In another embodiment of this system, the neural network architecture can include a binning classification network configured to predict a heat map of most probable values ​​for each ellipse parameter of the ellipse description.

[0022]

[0023] In another embodiment of this system, the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, where the abstraction can be employed to generate a heat map.

[0023]

[0024] In another embodiment of this system, the neural network architecture may comprise a recurrent network configured to extract features from the input video frames and directly predict values ​​characterizing each of the ellipse parameters.

[0024]

[0025] In another embodiment of the system, the animal can be a rodent.

[0026] In one embodiment, a non-transitory computer program product is provided having instructions stored thereon that, when executed by at least one data processor of at least one computing system, may perform a method including receiving video data representing observations of animals and implementing a neural network architecture. The neural network architecture may be configured to receive input video frames extracted from the video data; generate an ellipse description of at least one animal based on the input video frames, the ellipse description being defined by predetermined ellipse parameters; and provide, for the at least one animal, data including values ​​characterizing the predetermined ellipse parameters.

[0025]

[0027] In another embodiment, the ellipse parameters may be coordinates representing the animal's position within a plane, the length of the animal's major and minor axes, and the angle at which the animal's head is pointing, defined relative to the direction of the major axis.

[0026]

[0028] In another embodiment, the neural network architecture may be an encoder-decoder segmentation network configured to predict a foreground-background segmentation image from an input video frame, predict whether an animal is present in the input video frame based on the segmentation image in a pixel-wise manner, output a segmentation mask based on the pixel-wise prediction, and fit an ellipse to portions of the segmentation mask where an animal is predicted to be present to determine values ​​characterizing predetermined ellipse parameters.

[0027]

[0029] In another embodiment, the encoder-decoder segmentation network may include a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract an input video frame into a set of small spatial resolution features. The feature decoder may be configured to transform the set of features into a shape consistent with the input video frame and output a foreground-background segmented image. The angle predictor may be configured to predict an angle at which the animal's head is pointing.

[0028]

[0030] In another embodiment, the neural network architecture may comprise a binning classification network configured to predict a heat map of the most probable values ​​of each ellipse parameter of the ellipse description.

[0029]

[0031] In another embodiment, the binning classification network comprises a feature encoder configured to abstract input video frames to a smaller spatial resolution, the abstraction comprising: It can be employed to generate a tomography map.

[0030]

[0032] In another embodiment, the neural network architecture may comprise a recurrent network configured to extract features from the input video frames and directly predict values ​​characterizing each of the ellipse parameters.

[0031]

[0033] In another embodiment, the animal can be a rodent.

[0034] In one embodiment, a system is provided, which may include an arena and an acquisition system. The arena may include a frame and an enclosure attached to the frame. The enclosure may include a door sized to house an animal and configured to allow access to the interior. The acquisition system may include a camera, at least two sets of light sources, a controller, and a data storage device. Each set of light sources may be configured to emit light incident on the enclosure at a different wavelength. The camera may be configured to acquire video data of at least a portion of the enclosure when illuminated by at least one of the sets of light sources. The controller may be in electrical communication with the camera and the sets of light sources. The controller may be configured to generate control signals operative to control the acquisition of video data by the camera and the emission of light by the sets of light sources, and to receive the video data acquired by the camera. The data storage device may be in electrical communication with the controller and configured to store the video data received from the controller.

[0032]

[0035] In another embodiment of the system, at least a portion of the housing can be substantially opaque to visible light.

[0036] In another embodiment of the system, at least a portion of the housing may be formed from a material that is substantially opaque to visible light wavelengths.

[0033]

[0037] In another embodiment of the system, at least a portion of the housing may be formed from a material that is substantially non-reflective to infrared wavelengths.

[0038] In another embodiment of the system, at least a portion of the housing may be formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

[0034]

[0039] In another embodiment of this system, the first set of light sources may include one or more first illuminators configured to emit light at one or more visible light wavelengths, and the second set of light sources may include one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

[0035]

[0040] In another embodiment of this system, the wavelength of the infrared light can be about 940 nm.

[0041] In another embodiment of the system, the camera may be configured to acquire video data at a resolution of at least 480x480 pixels.

[0036]

[0042] In another embodiment of the system, the camera may be configured to capture video data at a frame rate that is higher than the frequency of mouse movements.

[0043] In another embodiment of the system, the camera may be configured to acquire video data at a frame rate of at least 29 frames per second (fps).

[0037]

[0044] In another embodiment of the system, the camera may be configured to acquire video data having at least an 8-bit depth.

[0045] In another embodiment of the system, the camera may be configured to acquire video data at infrared wavelengths.

[0038]

[0046] In another embodiment of the system, the controller may be configured to compress the video data received from the camera.

[0047] In another embodiment of this system, the controller may be configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing variance-based background subtraction.

[0039]

[0048] In another embodiment of this system, the filter for the MPEG codec can be Q0 HQDN3D.

[0049] In another embodiment of this system, the controller may be configured to request the first light source to illuminate the enclosure according to a schedule that simulates a light-dark cycle.

[0040]

[0050] In another embodiment of the system, the controller can be configured to request the first light source to illuminate the enclosure with visible light having an intensity of between approximately 50 lux and approximately 800 lux during the light portion of a light-dark cycle.

[0041]

[0051] In another embodiment of the system, the controller may be configured to request the second light source to illuminate the enclosure with infrared light such that the temperature rise of the enclosure due to the infrared illumination is less than 5°C.

[0042]

[0052] In another embodiment of this system, the controller may be configured to request the first light source to illuminate the enclosure according to 1024 logarithmically scaled levels of illumination.

[0043]

[0053] In one embodiment, a method is provided, the method may include illuminating an enclosure configured to house an animal with at least one set of light sources. Each set of light sources may be configured to emit light at a different wavelength from the others. The method may also include acquiring, with a camera, video data of at least a portion of the enclosure illuminated by at least one of the sets of light sources. The method may also include generating, with a controller in electrical communication with the camera and the sets of light sources, a control signal operative to control the acquisition of the video data by the camera and the emission of light by the sets of light sources. The method may further include receiving, with the controller, the video data acquired by the camera.

[0044]

[0054] In another embodiment of this method, at least a portion of the housing can be substantially opaque to visible light.

[0055] In another embodiment of this method, at least a portion of the housing may be formed from a material that is substantially opaque to visible light wavelengths.

[0045]

[0056] In another embodiment of this method, at least a portion of the housing may be formed from a material that is substantially non-reflective to infrared wavelengths.

[0057] In another embodiment of this method, at least a portion of the housing may be formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

[0046]

[0058] In another embodiment of this method, the first set of light sources may include one or more first illuminators configured to emit light at one or more visible light wavelengths, and the second set of light sources may include one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

[0047]

[0059] In another embodiment of this method, the wavelength of the infrared light can be about 940 nm.

[0060] In another embodiment of this method, the camera has a resolution of at least 480x480 pixels. The video data may be acquired at a resolution of 1080p.

[0048]

[0061] In another embodiment of this method, the camera may be configured to capture video data at a frame rate that is higher than the frequency of mouse movements.

[0062] In another embodiment of this method, the camera may be configured to acquire video data at a frame rate of at least 29 frames per second (fps).

[0049]

[0063] In another embodiment of this method, the camera may be configured to acquire video data having at least an 8-bit depth.

[0064] In another embodiment of this method, the camera may be configured to acquire video data at infrared wavelengths.

[0050]

[0065] In another embodiment of the method, the controller may be configured to compress the video data received from the camera.

[0066] In another embodiment of this method, the controller may be configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing variance-based background subtraction.

[0051]

[0067] In another embodiment of this method, the filter for the MPEG codec can be Q0 HQDN3D.

[0068] In another embodiment of this method, the controller may be configured to request the first light source to illuminate the enclosure according to a schedule that simulates a light-dark cycle.

[0052]

[0069] In another embodiment of this method, the controller may be configured to request the first light source to illuminate the enclosure with visible light having an intensity of between approximately 50 lux and approximately 800 lux during the light portion of a light-dark cycle.

[0053]

[0070] In another embodiment of this method, the controller may be configured to request the second light source to illuminate the enclosure with infrared light such that the temperature of the enclosure increases by less than 5°C due to the infrared illumination.

[0054]

[0071] In another embodiment of this method, the controller may be configured to request the first light source to illuminate the enclosure according to 1024 logarithmically scaled levels of illumination.

[0055]

[0072] These and other features will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0056] [Figure 1]

[0073] FIG. 1 is a flow diagram illustrating an exemplary embodiment of an operating environment for animal tracking. [Figure 2]

[0074] FIG. 2 is a schematic diagram of one embodiment of a system for animal activity monitoring. [Figure 3]

[0075] 3A-3F are images showing sample frames acquired by the system of FIG. 2; (A-C) visible light; (D-F) infrared (IR) light. [Figure 4]

[0076] 4A-4B are plots of quantum efficiency as a function of wavelength for two camera models; (A) relative response for the Sentech STC-MC33USB; (B) quantum efficiency of the Basler acA1300-60gm-NIR. [Figure 5]

[0077] FIG. 1 shows a plot of the transparency versus wavelength profile of an IR longpass filter. [Figure 6]

[0078] 6A-6D are images showing exemplary embodiments of video frames to which different compression techniques have been applied; (A) uncompressed; (B) MPEG4 Q0; (C) MPEG4 Q5; (D) MPEG4 Q0 HQDN3D; [Figure 7]

[0079] FIG. 7 illustrates an embodiment of components of an acquisition system suitable for use with the system of FIG. [Figure 8A]

[0080] FIG. 8A is a schematic diagram of an exemplary embodiment of an observation environment analyzed in accordance with the present disclosure, including a black mouse, a gray mouse, an albino mouse, and a mottled mouse. [Figure 8B]

[0081] FIG. 8B is a schematic diagram of a condition where animal tracking becomes insufficient. [Figure 8C]

[0082] FIG. 8C is a schematic diagram of an exemplary embodiment of mouse tracking including object tracking in the form of an ellipse. [Figure 9]

[0083] FIG. 9 is a schematic diagram of an exemplary embodiment of a segmentation network architecture. [Figure 10]

[0084] Figure 10 is a schematic diagram of an exemplary embodiment of a binning classification network architecture. [Figure 11]

[0085] Figure 11 is a schematic diagram of an exemplary embodiment of a regression classification network architecture. [Figure 12A]

[0086] FIG. 12A illustrates an exemplary embodiment of a graphical user interface showing the placement of two marks, one in the foreground (F) and one in the background (B). [Figure 12B]

[0087] FIG. 12B illustrates an example embodiment of a graphical user interface showing the segmentation resulting from the marking of FIG. 12A. [Figure 13A]

[0088] FIG. 13A shows plots of training curves for the segmentation, regression, and binning classification network embodiments of FIGS. [Figure 13B]

[0089] FIG. 13B shows plots of validation curves for the segmentation, regression, and binning classification network embodiments of FIGS. [Figure 13C]

[0090] Figure 13C shows plots of the training and validation performance of the segmentation network architecture of Figure 9. [Figure 13D]

[0091] Figure 13D shows a plot of the training and validation performance of the regression network architecture of Figure 11. [Figure 13E]

[0092] Figure 13E shows plots of the training and validation performance of the binned classification network architecture of Figure 10. [Figure 14A]

[0093] FIG. 14A illustrates a plot of training error as a function of training steps for sets of different sizes, according to an embodiment of the present disclosure. [Figure 14B]

[0094] FIG. 14B illustrates a plot of validation error as a function of training steps for sets of different sizes, according to an embodiment of the present disclosure. [Figure 14C]

[0095] FIG. 14C shows a plot of training and validation errors as a function of steps for the full training set of training samples. [Figure 14D]

[0096] FIG. 14D shows a plot of training and validation errors as a function of steps for a training set containing 10,000 (10k) training samples. [Figure 14E]

[0097] FIG. 14E shows a plot of training and validation errors as a function of steps for a training set containing 5,000 (5k) training samples. [Figure 14F]

[0098] FIG. 14F shows a plot of training and validation errors as a function of steps for a training set containing 2,500 (2.5k) training samples. [Figure 14G]

[0099] FIG. 14G shows a plot of training and validation errors as a function of steps for a training set containing 1,000 (1k) training samples. [Figure 14H]

[0100] FIG. 14H shows a plot of training and validation errors as a function of steps for a training set containing 500 training samples. [Figure 15]

[0101] 15A-15D are frames of captured video data overlaid with color indicators that distinguish each mouse from the others; (A-B) visible light illumination; (C-D) infrared light illumination. [Figure 16]

[0102] Figure 16 is a plot comparing the performance of the segmentation network architecture of Figure 9 with a beam-breaking system. [Figure 17]

[0103] FIG. 17A is a plot of predictions according to one embodiment of the present disclosure and Ctrax.

[0057]

[0104] Figure 17B shows the segmentation network architecture of Figure 9. 10 is a plot of the relative standard deviation of the short-axis predictions determined by the MRI scan. [Figure 18A]

[0105] Figure 18A is a plot of the total distance tracked for a large-scale strain study of genetically distinct animals determined by the segmentation network architecture of Figure 9. [Figure 18B]

[0106] Figure 18B is a plot of the circadian movement patterns observed in six animals continuously tracked over four days in a dynamic environment determined by the segmentation network architecture of Figure 9. DETAILED DESCRIPTION OF THE INVENTION

[0058]

[0107] Please note that the drawings are not necessarily to scale. They are intended to depict only representative aspects of the subject matter, and therefore should not be considered as limiting the scope of the disclosure.

[0059]

[0108] For clarity, in this specification, with respect to small rodents such as mice, one Exemplary embodiments of systems and corresponding methods for facilitating behavioral monitoring through video capture of animals or multiple animals and tracking of one or more animals are discussed, although the disclosed embodiments may be adapted and / or configured to monitor other animals as well, without limitation.

[0060]

[0109] FIG. 1 shows an arena 200, an acquisition system 700, and a neural network. FIG. 7 is a schematic diagram illustrating an exemplary embodiment of an operating environment 100 including a tracking system configured to implement a mouse tracker. As discussed in more detail below, one or more mice may be housed in an arena 200. Video data of at least one animal (e.g., a mouse) is acquired. The video data may be acquired alone or in combination with other data relevant to animal monitoring, such as audio and environmental parameters (e.g., temperature, humidity, light intensity). The process of acquiring this data, including control of cameras, microphones, lighting, other environmental sensors, data storage, and data compression, may be performed by an acquisition system 700. The acquired video data may be input to a tracking system capable of implementing a convolutional neural network (CNN) that tracks one or more animals based on the video data.

[0061] I. Video Data Acquisition

[0110] In one embodiment, a system for capturing video data including animal movements and As discussed below, video data is continuously acquired over a predetermined period of time (e.g., one or more minutes, one or more hours, one or more days, one or more weeks, one or more months, one or more years, etc.). The video data characteristics can be sufficient to facilitate subsequent analysis for the extraction of behavioral patterns, including, but not limited to, one or more of resolution, frame rate, and bit depth. A practical solution is provided and is expected to be more robust and of higher quality than existing video capture systems. Embodiments of the present disclosure are tested with multiple methods of visually marking mice. A practical example of synchronous acquisition of video and ultrasonic vocalization data is also presented.

[0062]

[0111] In one embodiment, animal monitoring is performed over a period of approximately 4 to 6 weeks. A video monitoring system for the arena may be deployed. Deployment may include one or more of image capture and arena design, fine-tuning of chamber design, development of video acquisition software, acquisition of audio data, stress testing of cameras, chambers, and software, and determination of chamber production for the deployment phase. Each of these is described in detail below. The aforementioned four to six week observation period is provided for illustrative purposes, and it will be understood that embodiments of the present disclosure may be employed for longer or shorter periods as needed.

[0063] a. Arena design

[0112] Appropriate arena design can be important for obtaining high-quality behavioral data. The arena is the animal's "home" and may be configured to provide one or more of isolation from environmental disturbances, proper circadian lighting, food, water, bedding, and is generally a stress-free environment.

[0064]

[0113] From a behavioral perspective, the arena minimizes stress and environmental disturbances. It would also be desirable to be able to display natural behavior.

[0114] From a husbandry perspective, the arena is designed for easy access to cleaning, adding or removing mice, retrieving food, and feeding. And it would be desirable to facilitate the addition and removal of water.

[0065]

[0115] From a veterinary perspective, the arena provides a healthy environment without substantially inhibiting behavioral interest. It would be desirable to facilitate the provision of diagnosis and treatment as well as the monitoring of environmental conditions (eg, temperature, humidity, light, etc.).

[0066]

[0116] From a computer vision perspective, the arena is a highly immersive experience, with substantial occlusion, distortion, and It would be desirable to facilitate the acquisition of high quality video and audio without interference from noise, reflections, and / or noise pollution, and without substantially interfering with the presentation of the activity of interest.

[0067]

[0117] From an amenity standpoint, the arena will be designed to minimize floor space and It would be desirable to provide relatively easy storage without the need for disassembly or reassembly.

[0068]

[0118] Therefore, the arena provides a balance of behavior, husbandry, computation, and facilities. An exemplary embodiment of an arena 200 is shown in FIG. 2. The arena 200 may include a frame 202 to which a housing 204 is mounted. The housing 204 may include a door 206 configured to allow access to the interior. One or more cameras 210 and / or lights 212 may be mounted adjacent to the frame 202 (e.g., above the housing 204) or directly to the frame 202.

[0069]

[0119] As discussed in more detail below, in certain embodiments, the illumination 212 may include at least The housing 204 may include at least two sets of light sources, each set of light sources may include one or more luminaires configured to emit light incident on the housing 204 at a different wavelength than the other set of light sources. A first set of light sources may be configured to emit light at one or more visible wavelengths (e.g., from approximately 390 nm to approximately 700 nm), and a second set of light sources may be configured to emit light at one or more infrared (IR) wavelengths (e.g., greater than approximately 700 nm to approximately 1 mm).

[0070]

[0120] The camera 210 and / or lighting 212 may be connected to a user interface 214. The user interface 214 may be a display configured to display video data captured by the camera 210. In certain embodiments, the user interface 214 may be a touchscreen display configured to display one or more user interfaces for controlling the camera 210 and / or the light 212.

[0071]

[0121] Alternatively or additionally to the above, a camera 110, lighting 212, and user input The interface 214 may be in electrical communication with a controller 216. The controller 216 may be configured to generate control signals that operate to control the acquisition of video data by the camera 210, the emission of light by the illumination 212, and / or the display of the acquired video data by the user interface 214. In certain embodiments, the user interface may optionally be omitted.

[0072]

[0122] The controller 216 may also be in communication with a data storage device 220. The controller 216 may be configured to receive video data captured by the camera 210 and transmit the captured video data to the data storage device 220 for storage. Communication between one or more of the camera 210, the lighting 212, the user interface 214, the controller 216, and the data storage device 220 may be performed using wired communication links, wireless communication links, and combinations thereof.

[0073]

[0123] As discussed below, the arena 200 is a facility for the behavior, husbandry, computation, and equipment. It may have an open field design configured to achieve the desired balance while allowing for completion in a predetermined period of time (eg, approximately 5 months).

[0074] material

[0124] In certain embodiments, the housing 204 (e.g., the lower portion of the housing 204) is configured At least a portion of the material comprising the housing 204 may be substantially opaque to visible light wavelengths. In this manner, visible light emitted by sources other than the light 212, as well as visual cues observable by an animal within the housing 204 (e.g., movement of objects and / or the user), may be reduced and / or substantially eliminated. In additional embodiments, the material comprising the housing 204 may be substantially non-reflective to infrared wavelengths to facilitate video data acquisition. The wall thickness of the housing 204 may be selected within a range suitable for providing mechanical support (e.g., approximately 1 / 8 inch to approximately 1 / 4 inch).

[0075]

[0125] In one embodiment, the housing 204 is made of polyvinyl chloride (PVC) or polyvinyl chloride (PVC). The arena 200 may be constructed using a foam sheet formed of polydimethylsiloxane (POM). One example of a POM is Delrin® (DuPont, Wilmington, DE, USA). Such a foam sheet is advantageous because it may provide sufficient versatility and durability for long-term animal monitoring of the arena 200.

[0076]

[0126] In one embodiment, the frame 202 has a plurality of legs 202a and a and one or more shelves 202b extending horizontally (e.g., horizontally). By way of example, the frame 202 can be a commercially available shelving system of a predetermined size with fixed wheels for transportation to a storage area. In one embodiment, the predetermined size is approximately 61 cm. m (2 ft) x 61 cm (2 ft) x 183 cm (6 ft) (e.g., Super Erecta Metroseal 3™, InterMetro Industries Corporation, Wilkes-Barre, PA, USA), although arenas of different sizes may be employed without limitation in other embodiments.

[0077] b. Data Acquisition

[0127] The video capture system includes a camera 210, lighting 212, and a user interface. 214, a controller 216, and a data storage device 220. The video capture system may be adapted to have a predetermined balance of performance characteristics, including, but not limited to, one or more of the frame rate of the video capture, bit depth, resolution of each frame, and spectral sensitivity in the infrared region, as well as video compression and storage. As discussed below, these parameters may be optimized to maximize the quality and minimize the quantity of data.

[0078]

[0128] In one embodiment, the camera 210 has a resolution of approximately 640 x 480 pixels, The camera 210 may capture video data having at least one of approximately 29 fps and approximately 8-bit depth. Using these video capture parameters, approximately 33 GB / hour of uncompressed video data may be generated. As an example, the camera 210 may be a Sentech USB2 (Sensor Technologies America, Inc., Carrollton, TX, USA). Figures 3A-3F show sample frames captured from one embodiment of a video capture system using visible light (Figures 3A-3C) and infrared (IR) light (Figures 3D-3F).

[0079]

[0129] As discussed below, the collected video data may be transmitted to the camera 210 and / or Alternatively, the signal may be compressed by the controller 216.

[0130] In another embodiment, the video capture system adjusts the resolution of the captured video data to It can be configured to approximately double the resolution (e.g., approximately 960x960 pixels). Four other cameras with higher resolution than the Sentech USB were investigated, as shown below.

[0080] [Table 1]

[0081]

[0131] These cameras range in cost, resolution, maximum frame rate, bit depth, and They may differ in terms of their photoluminescence and quantum efficiency.

[0132] An embodiment of the video capture system is monochrome, approximately 30 fps, and It can be configured to collect 8-bit depth video data. According to the Shannon-Nyquist theorem, the frame rate should be at least twice the frequency of the events of interest (see, e.g., Shannon (1994)). Mouse behavior can vary from a few hertz for grooming to 20 hertz for rapid movements (see, e.g., Deschenes et al. (See also Kalueff et al. (2012), Kalueff et al. (2010), and Wiltschko et al. (2015)). Grooming has been observed to occur at rates up to approximately 7 Hz. Therefore, for observing most mouse behaviors, it may be appropriate to record video at a frame rate higher than the frequency of mouse movements (e.g., approximately 29 fps). However, cameras can rapidly lose sensitivity in the IR range. This loss of contrast can be overcome by increasing the level of IR light, although increasing the intensity of IR light can increase the environmental temperature.

[0082] illumination

[0133] As mentioned above, the illumination 212 may be one or more types of light, such as visible white light and infrared light. The illumination may be configured to emit a variety of light. Visible light may be employed for illumination and may be programmed (e.g., by the controller 216) to provide a light-dark cycle and adjustable intensity. The ability to adjust the lighting cycle allows for simulating the light from the sun that animals experience in the wild. The length of the light-dark period may be adjusted to simulate seasons, and lighting shifts may be implemented to simulate jet lag (circadian phase advance and delay) experiments. Also, high-intensity lighting may be employed to induce anxiety in certain animals, while low-intensity lighting may be employed to elicit different exploratory behaviors. Thus, the ability to temporally control the length of light and dark periods and light intensity is essential for proper behavioral experiments.

[0083]

[0134] In certain embodiments, the controller 216 controls the temperature during the light portion of the light-dark cycle. The housing 204 is irradiated with visible light having an intensity of approximately 50 lux to approximately 800 lux. The light source may be configured to require a light intensity of approximately 200 lux to approximately 300 lux. The selected light intensity may vary depending on the type of movement to be observed. In one embodiment, a relatively low light intensity (e.g., approximately 200 lux to approximately 300 lux) may be employed to encourage and observe exploratory movement by the mouse.

[0084]

[0135] In a particular embodiment, an IR long-pass filter is used to reduce the I In the R domain, substantially all video data may be captured by the camera 210. An IR long-pass filter may eliminate substantially all visible light input to the camera 210. IR light is beneficial because it allows for uniform illumination of the enclosure 104, day or night.

[0085]

[0136] Two wavelengths of IR light were evaluated (850 nm and 940 nm LEDs). 850 nm light produces a vivid red hue visible to the naked eye and can provide low-intensity exposure to animals, but such dim light can cause mood swings in mice. Therefore, 940 nm light is chosen for recording.

[0086]

[0137] Recording at a wavelength of 940 nm can be difficult because the quantum yield in the camera can be very low. , which can result in grainy-looking images due to high gain. Therefore, various infrared illumination levels using different cameras were evaluated to identify the maximum light level that could be achieved without substantially increasing the temperature of the housing 204 due to infrared illumination. In certain embodiments, the temperature of the housing 204 can be increased by approximately 5° C. or less (e.g., approximately 3° C. or less).

[0087]

[0138] The Basler acA1300-60gm-NIR camera was also evaluated. This camera has approximately 3-4 times the spectral sensitivity at 940 nm compared to the other cameras listed in Table 1, as shown in Figures 4A and 4B. Figure 4A shows the spectral sensitivity of the Sentech camera as a representative example in terms of relative response, and Figure 4B shows the spectral sensitivity of the Basler camera in terms of quantum efficiency. Quantum efficiency is a measure of electrons emitted per photon that strikes the sensor. Relative response is quantum efficiency expressed on a scale of 0 to 1. For reference, the wavelength of 940 nm is also shown as a vertical line in Figures 4A and 4B.

[0088]

[0139] The visible light cycle provided by the light 212 is controlled by the controller 216 or The visible light may be controlled by another device in communication with the light 212. In a particular embodiment, the controller 216 may include a lighting control panel (Phenome Technologies, Skokie, IL). The control panel has 1024 levels of lighting that are logarithmically scaled, controllable via an RS485 interface, and capable of implementing dawn / dusk events. As discussed in more detail below, control of the visible light may be incorporated into control software executed by the controller 216.

[0089] filter

[0140] As mentioned above, optionally, substantially all visible light is filtered out from the camera during video data acquisition. An IR long-pass filter may be employed to prevent IR from reaching camera 210. As an example, a physical IR long-pass filter may be employed with camera 110. This configuration may provide substantially uniform illumination regardless of the light or dark phase of arena 200.

[0090]

[0141] Filters Potentially Suitable for Use in Embodiments of the Disclosed Systems and Methods Profiles are shown in Figure 5 (e.g., IR pass filters 092 and 093). IR cut filter 486, which blocks IR light, is shown for comparison. Additional profiles for RG-850 (glass, Edmunds Optics) and 43-949 (plastic, laser curable, Edmunds Optics) may also be suitable.

[0091] lens

[0142] In one embodiment, the camera lens has a 0.847 cm (1 / 3") A 3.5-8mm, f1.4 (CS mount) is possible. This lens would produce the images seen in Figures 3A and 3B. A C-mount lens analogue could also be employed.

[0092] Video Compression

[0143] Ignoring compression, the camera 210 uses approximately 1MB per frame, or approximately Raw video data may be generated at a rate of 30 MB / sec, approximately 108 GB / hour, or approximately 2.6 TB / day. A variety of objectives may be considered when selecting a storage method. Depending on the context of the video, removing certain elements of the video before long-term storage may be a useful option. Also, when considering long-term storage, applying a filter or other form of processing (e.g., by controller 216) may be desirable. However, if the processing method is to be changed later, preserving the original or raw video data may be a useful solution. An example of a video compression test is described below.

[0093]

[0144] Pixel resolution approximately 480x480, approximately 29 fps, and approximately 8 bits Several compression standards were evaluated on video data collected over approximately 100 minutes at 1000p / pixel. The two lossless formats tested from raw video were Dirac and H264. H264 offers slightly smaller file sizes but requires slightly longer transcoding times. Dirac may be more widely supported due to subsequent transcoding to another format.

[0094]

[0145] The MPEG4 lossy format was also evaluated, as it is closely related to H264. It is known that bitrate can be well controlled. There are two methods for setting the bitrate. The first is to set a constant bitrate throughout the entire encoded video, and the second is to set a variable bitrate based on the deviation from the original video. In ffmpeg using the MPEG4 encoder, variable bitrate setting can be easily achieved by selecting a quality value (0 to 31, where 0 is nearly lossless).

[0095]

[0146] In Figures 6A-6D, three different image compression methods are compared on original (raw) captured video frames. 6 The other three methods are shown in Fig. 6 B~Figure 6 In D, the difference in pixels from the original image shows only the effect of compression. That is, the compressed image is this different from the original. Therefore, small differences are better, and high compression ratios are better. 6 As shown in Figure B, the compression performed according to the MPEG4 codec with the Q0 filter shows a compression ratio of 1 / 17. 6 As shown in Fig. C, the compression performed according to the MPEG4 codec with the Q5 filter shows a compression ratio of 1 / 237. 6 As shown in D, the compression performed according to the MPEG4 codec with the HQDN3D filter shows a compression ratio of 1 / 97.

[0096]

[0147] The video data collected in accordance with the disclosed embodiments is filtered using a quality 0 parameter (Q0 filter (Figure 1)). 6 B), Q0 HQDN3D filter (Fig. 6 D)), approximately 0.01% of the pixels change from the original image (up to 4% increase or decrease in intensity). This accounts for roughly 25 pixels per frame. Most of these pixels are located at the shadow boundaries. Naturally, this small image change is subject to the magnitude of noise interfering with the camera 210 itself. Higher quality values ​​(e.g., Q5 (Figure 1)) can be achieved by using a higher quality value (e.g., Q6 (Figure 1)). 6 In C), in order to better compress the video data, artifacts may be introduced. These often lead to blocky artifacts that appear if care is not taken during compression.

[0097]

[0148] In addition to these formats, to accommodate individual user datasets, Other suitable lossless formats can be generated. Two of these include the FMF codec (fly movie format) and the UFMF codec (micro fly movie format). The purpose of these formats is to minimize extraneous information while optimizing readability for tracking. Because these formats are lossless and operate on a fixed background model, unfiltered sensor noise did not allow for any substantial data compression. The results of this compression evaluation are shown in Table 2.

[0098] [Table 2]

[0099]

[0149] In addition to the selection of a codec for data compression, it is also desirable to reduce background noise in the image. Background noise is inherent in all cameras and is often referred to as dark noise, and represents the baseline noise in an image.

[0100]

[0150] To eliminate this noise, increase the exposure time, increase the aperture, and decrease the gain. However, these methods are not viable options if they directly affect the experiment. Therefore, the HQDN3D filter in ffmpeg can be adopted, which incorporates spatiotemporal information and removes small fluctuations.

[0101]

[0151] As shown in Figures 6B to 6D, the HQDN3D filter It is observed that the file size of the data is significantly reduced (e.g., about 100 times smaller than the file size of the original video data). After compression with the G4 codec, the resulting average bitrate is approximately 0.34 GB / hour for compressed video. Furthermore, it has been experimentally verified that virtually all information loss is several orders of magnitude less than that resulting from sensor noise (video acquired without a mouse). This kind of noise removal significantly improves compressibility.

[0102]

[0152] Unexpectedly, the HQDN3D filter is a convolutional neural network filter, which is discussed in detail below. It has been found that the HQDN3D filter significantly improves the performance of neural network (CNN) tracking. Without being bound by theoretical constraints, we believe this improvement is achieved because the HQDN3D filter is a variance-based background subtraction method. With low variance, it is easier to identify the foreground, resulting in high-quality tracking.

[0103] Ultrasound Audio Acquisition

[0153] Mice use ultrasonic vocalizations for social communication, mating, and They may engage in aggression, and rearing (see, for example, Grimsley et al. (2011)). Combined with olfactory and tactile cues, vocalizations may be one of the most prominent forms of communication in mice. Although not tested in mice, in humans, changes in voice and vocalizations (aging) may define transitions such as puberty and aging (see, for example, Decoster and Debruyne (1997), Martins et al. (2014), and Mueller (1997)).

[0104]

[0154] Thus, as discussed in more detail below, embodiments of arena 200 include: The frame 202 may further include one or more microphones 222. The microphones 222 may be attached to the frame 202 and configured to acquire audio data from the animals contained in the enclosure 204. Using the microphones 222 in the form of a microphone array may lead to synchronous data collection. This configuration of microphones 222 allows for the identification of vocalizing mice. The ability to further determine vocalizing mice within a group of mice has recently been demonstrated using microphone arrays (see, e.g., Heckman et al. (2017), Neunuebel et al. (2015)).

[0105]

[0155] The data collection setup is provided similar to that of Neunuebel et al. Four microphones can be positioned on the sides of the arena that can capture sound. When integrated with video data, vocalizing mice can be identified using maximum likelihood methods (see, e.g., Zhang et al. (2008)).

[0106] Environmental Sensors

[0156] In one embodiment, the arena 200 is configured to control temperature, humidity, and / or light intensity. The controller 216 may further include one or more environmental sensors 224 configured to measure one or more environmental parameters, such as light (e.g., visible and / or IR). In certain embodiments, the environmental sensors 224 may be integrated and configured to measure more than one environmental parameter (see, e.g., Phenome Technologies, Skokie, IL). The environmental sensors 224 may be in electrical communication with the controller 216 to collect daily temperature and humidity data along with light levels. The collected environmental data may be output for display in a user interface indicating minimum and maximum temperatures as well as lighting conditions (see the description of control software below).

[0107] Software Control System

[0157] Software controlled by controller 216 for data acquisition and light control The system may be implemented in a software control system that may be configured to independently collect video, audio / ultrasound, and environmental data along with corresponding timestamps. In this manner, the system may be configured to capture video, audio / ultrasound, and environmental data for any predetermined period of time (e.g., one or more seconds, Data may be collected uninterrupted over a period of one or more minutes, one or more hours, one or more days, one or more years, etc. This may allow for later compilation or synchronization of the acquired video, audio / ultrasound, and / or environmental data for analysis or presentation.

[0108] operating system

[0158] The choice of operating system depends on the availability of drivers for various sensors. For example, only the Avisoft Ultrasonic microphone driver is compatible with Windows operating systems. However, this choice may affect:

[0109] Inter-process communication: Inter-process communication options are influenced by the underlying OS. Similarly, the OS influences the choice of communication between threads. However, developing on a cross-platform framework such as QT can bridge the gap.

[0110] Accessing the System Clock: The method for accessing the high-resolution system clock varies from OS to OS, as discussed in more detail below. Hardware Options

[0159] In certain embodiments, the control system is in the form of a single board computer. The controller 216 may be implemented in a variety of ways. Several options are available, including military-grade / industrial computers that are highly ruggedized for continuous operation.

[0111] External Clock vs. System Clock

[0160] The system clock can be used to generate the appropriate A real-time clock value is available. In Posix systems, the clock_gettime(CLOCK_MONOTONIC,...) function can return seconds and nanoseconds. The clock resolution can be queried with the clock_getres() function. It would be desirable for the clock resolution of a control system embodiment to be less than the frame period of approximately 33 milliseconds. In one embodiment, the system clock is a Unix system.

[0112]

[0161] GetTickCore, used to get the number of milliseconds since the system started The unt64() system function has been developed. The expected resolution of this timer is approximately 10 to approximately 16 milliseconds. It can serve the same purpose as the clock_gettime() system call, although it can be useful to check and take into account wrapping of the value.

[0113]

[0162] On a Macintosh computer, you can access the system clock in the same way. The following code snippet has been evaluated and sub-microsecond resolution has been observed:

[0114] clock_serv_t cclock; mach_timespec_t mts; host_get_clock_service(mach_host_self(),SYSTEM_CLOCK,&cclock); clock_get_time(cclock,&mts);

[0163] In any OS, system calls that return the time are adjusted periodically. If the time is increased, it may move backward. In one embodiment, a monotonically increasing system clock may be employed. GetTickCount64(), clock_gettime(), and clock_get_time() may all meet this criteria.

[0115] Video File Size

[0164] The camera supplier's software automatically splits the image into reasonable sizes. It is unlikely that the controller 116 will save a time-stamped output file. It is desirable for an embodiment of the controller 116 to collect video data without interruption, read each frame from the camera 110, and provide the collected video data in a simple form. For example, the controller 116 may be configured to provide approximately 10 minutes of video frames per file to the data storage device 120 in raw format, with timestamp headers or timestamps between frames, and each file would be under 2 GB.

[0116] Control System Architecture

[0165] FIG. 7 is a block diagram illustrating components of an acquisition system 700. In an embodiment, the acquisition system 700 may be executed by the controller 216. Each block represents a separate process or thread of execution. Controller Process

[0166] A control process is configured to start and stop other processes or threads. The control process may also be configured to provide a user interface for the acquisition system 700. The control process may be configured to keep a log of activities and may record (e.g., in a log) errors that occur during acquisition. The control process may also be configured to resume paused processes or threads.

[0117]

[0167] The method of communication between components can be determined after the system OS is selected. The user interface to the process can be a command line interface or a graphical interface, which can be built on a portable framework such as QT, which provides OS independence.

[0118] Video Acquisition Process

[0168] The video acquisition process communicates directly with the camera 210 to capture time-stamped The video capture process may be configured to save frames to data storage 220. The video capture process may operate at a high priority to minimize the chance of dropped frames. The video capture process may be kept relatively simple by minimizing processing between frames. The video capture process may also be configured to ensure proper exposure with the minimum effective shutter speed by controlling the IR illumination emitted by illumination 212.

[0119] Audio Acquisition Process

[0169] A separate audio acquisition process captures the ultrasound audio along with the appropriate time stamps. The audio system may be configured to acquire audio data. In one embodiment, the audio system may include an array of microphones 222 disposed in audio communication with the housing 204. In particular embodiments, one or more of the microphones 222 may be positioned within the housing 204. Each microphone in the microphone array may have one or more of the following capabilities: a sampling frequency of approximately 500 kHz, an ADC resolution of approximately 16 bits, a frequency range of approximately 10 kHz to approximately 20 kHz, and an 8th-order, 210 kHz anti-aliasing filter. By way of example, each microphone in the microphone array may include a Pettersson M500 microphone (Pettersson Elektronik AB, Uppsala, Sweden) or a functional equivalent. As described above, audio data captured by the microphones 222 may be time-stamped and provided to the controller 216 for analysis and / or to the data storage device 220 for storage.

[0120] Environmental Data Acquisition Process

[0170] A separate environmental data acquisition process collects environmental data such as temperature, humidity, and light levels. The environmental data may be collected at a low frequency (e.g., approximately 0.01 Hz to 0.1 Hz). The environmental data may be stored by the data storage device 220 (e.g., as one or more CSV files) with a timestamp for each record.

[0121] Lighting Control Process

[0171] The lighting control process controls the lighting 212 to provide a day-night cycle for the mice. In one embodiment, as described above, the camera 210 may be configured to filter out substantially all visible light and respond only to IR, a process that can avoid impacting video capture because the visible light can be filtered so that no IR occurs.

[0122] Video Editing Process

[0172] The video editing process involves converting the captured video data into a predefined compressed format. It can be configured to repackage the video into a predefined format. This process can be kept separate from the video acquisition to minimize the chance of dropped frames. The video editing process can run as a low priority background task or after data acquisition is complete.

[0123] Watchdog Process

[0173] A watchdog process monitors the health of the data acquisition process. The watchdog process may be configured to monitor the component. As an example, it may record the problem (e.g., in a log) and cause a restart if necessary. The watchdog process may also listen for a "heartbeat" from the component it is monitoring. Typically, a heartbeat may be a signal sent to the controller 216 to confirm that the component of the system 700 is operating normally. As an example, if a component of the system 700 stops functioning, the controller 216 may detect that the component is not sending a heartbeat. After this detection, the controller 216 may record the event and issue an alarm. Such alarms may include, but are not limited to, audio and visual alarms (e.g., lights, alphanumeric displays, etc.). Alternatively or in addition to such an alarm, the controller 216 may attempt to restart the component's operation, such as by sending a reinitialization signal or toggling power. The method of communication between the components of the system 700 and the controller 216 may vary depending on the OS choice.

[0124] Mouse markings

[0174] In certain embodiments, the mouse is marked to facilitate tracking. However, as discussed in more detail below, marking may be omitted and tracking facilitated by other techniques.

[0125]

[0175] Mouse marking for visual identification requires multiple non-trivial parameters In one embodiment, mice can be marked for long periods (weeks) that are invisible to the mice themselves and thus have minimal impact on their communication and behavior. As an example, long-term IR-sensitive markers that are invisible to the normal mouse visual field can be employed.

[0126]

[0176] In an alternative embodiment, human hair color and hair bleach are used to The mice's fur can be marked. This technique allows mice to be clearly identified over several weeks and can be used successfully in behavioral experiments (see, e.g., Ohayon et al. (2013). However, the process of marking the hairs requires anesthetizing the mice, which is an unacceptable process for this mouse monitoring system. Anesthesia alters physiology, and the hair dye itself can be an irritant that often alters mouse behavior. Because each DO mouse is unique, this can introduce unknown variables into the dye / anesthesia × genotype effect.

[0127]

[0177] Yet another method is being adopted using IR dye-based markers and tattoos. can be optimized.

[0178] In another embodiment, a pattern is placed on the back of the mouse as a form of marking. Shaving may be employed to create

[0128] Data Storage

[0179] During the development stage, a total of less than 2TB of data may be required. The data may include sample raw and compressed video from a variety of cameras and compression methods. Thus, data transfer of integrated USV and video data, as well as long-term video data of up to 7-10 days during stress testing, may be achieved. The size of the video may be reduced according to the selected compression standard. Sample data storage estimates are given below. test: One Arena Up to 5 cameras Video duration: Approximately 1-2 hours each Total approximately 10GB (maximum) Load testing: One Arena One camera Video duration: 14 days Resolution: 2x the current resolution (960 x 960) Total approximately 2TB production: 120 total runs (12–16 arenas, 80 animals per group run, alternating experiments) Duration (each): 7 days Resolution: 2x the current resolution (960 x 960) 32.25TB II. Animal Tracking

[0180] Video tracking of animals such as mice can be complex and time-consuming without a high level of user involvement. Existing animal monitoring systems are not feasible in dynamic environments or for genetically heterogeneous animals, making large-scale experiments unfeasible. As discussed below, attempts to track many different mouse strains in multiple environments using existing systems and methods reveal that these systems and methods are inadequate for large-scale experimental data sets.

[0129]

[0181] They come in different colors, including black, agouti, albino, gray, brown, nude, and mottled. An exemplary dataset containing mice of various coat colors was used for analysis. All animals were tested according to JAX-IACUC procedures outlined below. Mice were tested between 8 and 14 weeks of age. The dataset included 1,857 videos from 59 strains, totaling 1,702 hours.

[0130]

[0182] All animals were sourced from the Jackson Laboratory production colony. Behaviorally monitor adult mice aged 8–14 weeks according to certified procedures according to the Institute's Institutional Animal Care and Use Committee guidelines. were tested. Open-field behavioral assays were performed as described by Kumar (2011). Briefly, group-housed mice were weighed and habituated to the testing room for 30–45 min before the start of video recording. The first 55 min of locomotion data are presented herein. When available, eight males and eight females from each inbred and F1 isogenic strain were tested.

[0131]

[0183] In one embodiment, mice were placed in the same open field apparatus (e.g., It would be desirable to track multiple animals in a video capture system (arena 200). Examples of full-frame and cropped video images captured by the video capture system are shown in columns 1 (full frame) and 2 (crop) of FIG. 8A. Examples of ideal and actual tracking frames are shown in each environment for different genetic backgrounds (columns 3 (ideal tracking) and 4 (actual tracking) of FIG. 8A).

[0132]

[0184] In another aspect, an embodiment of the arena 200 includes food and water bowls and It would be desirable to perform video analysis of behavior in harsh environments, such as the Knockout Mouse Project (KOMP2) at the Jackson Laboratory (Figure 8A, columns 5 and 6, respectively).

[0133]

[0185] In the 24-hour apparatus, mice were placed in an ant enclosure with a blank bedding and food / water bowls. Mice were housed in arena 200. Mice were restrained in arena 200 and continuous recordings were performed under day-night conditions using infrared light emitted by light 212. Bedding and food bowls were moved by the mice and the visible light emitted by light 212 was changed over the course of each day to mimic a day-night cycle.

[0134]

[0186] The KOMP2 project collected data over a five-year period, but the beam It was desirable to perform video-based recordings as an additional mode of analysis to detect gait effects that could not be identified with the break system. In gait analysis, the animal's movements are analyzed. If the animal's gait is abnormal, skeletal, muscular, and / or neurological abnormalities can be derived. The KOMP2 project uses a beam-break system in which mice are placed in transparent polycarbonate boxes that are fully illuminated with infrared light. The floor of the matrix is ​​also polycarbonate, and the underlying bench surface is dark gray. Several boxes placed at the junction of two tables allow for jointing, and overhead lighting (e.g., LED lighting) can provide a unique high brightness for all boxes.

[0135]

[0187] In one embodiment, a background subtraction and speckle detection heuristic is used. Video tracking of this dataset was attempted using Ctrax, a modern open-source tracking tool. Ctrax abstracts the mouse for each frame along five metrics: the major and minor axes, the x and y positions of the mouse's center, and the animal's orientation (Branson, 2009). It also utilizes the MOG2 background subtraction model, where the software estimates both the mean and variance of the video background used for background subtraction. Ctrax uses the predicted foreground shape to fit an ellipse.

[0136]

[0188] In another embodiment, commercially available tracking software using a proprietary tracking algorithm is used. Video tracking of this dataset was attempted using LimeLight, a software that uses a single keyframe background model for segmentation and detection. Once a mouse is detected, LimeLight uses a proprietary algorithm to abstract the mouse to its center of gravity.

[0137]

[0189] This dataset poses significant challenges for these existing analysis systems. For example, Ctrax and LimeLight allow mice to differentiate between coat color and environmental combinations. It was difficult to handle such a combination. Generally, environments exhibiting high contrast, such as dark mice (e.g., black, agouti) on a white background, produced good tracking results. However, environments exhibiting low contrast, such as light mice (e.g., albino, gray, or spotted mice) on a white background, produced poor results. A black mouse in a white open field exhibited high foreground-background contrast, resulting in actual tracking closely matching the ideal. Gray mice visually resemble the arena walls, often obscuring their noses when facing away from them. Albino mice often remain hidden during tracking due to their similarity to the arena background. Spotted mice split in half due to their patterned fur. Although attempts were made to optimize and fine-tune Ctrax for each video, a significant number of poorly tracked frames were still observed, as shown in the actual tracking results in the fourth column (actual tracking) compared to the third column (ideal tracking) in Figure 8A. Discarding bad tracking frames is undesirable as it can bias the sampling and distort the biological interpretation.

[0138]

[0190] These errors are best tracked in environments such as 24-hour environments and KMOP2 environments. We observed that the error increases when the mouse is no longer the target. Furthermore, the distribution of errors is not random. For example, as shown in the fourth column of Figure 8 (actual tracking), tracking is extremely inaccurate when the mouse is in a corner, near a wall, or on top of the food bowl, whereas tracking is less inaccurate when the mouse is in the center. Placing a food bowl in the arena in a 24-hour environment can cause tracking problems if the mouse climbs on it. Arenas with reflective surfaces, such as KOMP2, also introduce errors into the tracking algorithm.

[0139]

[0191] Further investigation into the cause of improper tracking revealed that in most cases, improper tracking is caused by the mouse We found that these problems were due to insufficient segmentation of the mouse from the background. This included cases where the mouse was removed from the foreground or where the background was subsumed into the foreground due to insufficient contrast. Previously, some of these hurdles were addressed by altering the environment for optimized video data collection. For example, to track an albino mouse, the background color of the open field could be changed to black to increase contrast. However, such environmental alterations are not appropriate in this context. Because environmental color affects mouse and human behavior, such manipulations could potentially confound experimental results (Valdez, 1994; Kulesskaya, 2014). Furthermore, in a 24-hour data collection system or KOMP2 arena, such solutions may not work for spotted mice.

[0140]

[0192] Ctrax uses a single background model algorithm, so other background models are not supported. Tests were conducted to determine whether segmentation algorithms could improve tracking results. 26 different segmentation algorithms (Sobral (2013)) were tested, and it was discovered that each of these conventional algorithms performed well in certain situations and failed in others, as shown in Figure 8B. Other available systems and methods for animal tracking rely on background subtraction techniques for tracking. Since all 26 background subtraction methods failed, the results of Ctrax and LimeLight are believed to be representative of these other techniques. These segmentation algorithms are believed to fail due to improper segmentation.

[0141]

[0193] As such, there are many tracking solutions for analyzing video data. However, representative existing solutions have not been successful in overcoming the fundamental problem of proper mouse segmentation to achieve high-quality mouse tracking. None adequately address the fundamental problem of mouse segmentation and rely largely on environmental optimization to achieve proper segmentation, creating potential confusion.

[0142]

[0194] Furthermore, the time cost of fine-tuning the parameters of the background subtraction algorithm is prohibitive. For example, in tracking data over a 24-hour period, if a mouse sleeps in the same position for an extended period of time, it becomes part of the background model and becomes untrackable. Typical monitoring involves an experienced user interacting with the mouse for five minutes for each hour of video to ensure high-quality tracking results. While this level of user interaction is manageable for small, limited experiments, larger, longer-term experiments require extended involvement to monitor tracking performance.

[0143]

[0195] Embodiments of the present disclosure overcome these difficulties and allow for the detection of video data containing animals such as mice. We will build a robust next-generation tracker suitable for analyzing animal data. As discussed in detail below, we will employ an artificial neural network that achieves high performance under complex and dynamic environmental conditions, is independent of coat color genetics, and does not require continuous fine-tuning by the user.

[0144]

[0196] Convolutional neural networks learn to represent data at multiple levels of abstraction. These methods have dramatically improved state-of-the-art speech recognition, visual object recognition, object detection, and many other areas, such as drug discovery and genomics (LeCun (2015)). One advantage is that once an efficient network with suitable hyperparameters is developed, the neural network can be easily extended to other tasks by simply adding appropriate training data. Thus, the disclosed embodiments provide a highly generalizable solution for mouse tracking. Neural Network Architecture

[0197] Three main network architectures have been developed to solve the visual tracking problem: In one embodiment, object tracking may take the form of an elliptical description of the mouse based on a segmentation mask, as shown in Figure 8C (see Branson (2005)). In alternative embodiments, shapes other than an ellipse may be employed.

[0145]

[0198] The ellipse representation is defined by six variables, also referred to herein as parameters: The variables may describe the location of the animal. In one embodiment, one of the variables may be a coordinate defining a position in a predetermined coordinate system (e.g., x and y in a Cartesian coordinate system) representing the pixel location (e.g., mean center location) of the mouse in the acquired video frame. That is, a unique pixel location within a plane. Optionally, if necessary, landmarks (e.g., corners of the housing 204) may be detected in the video frame to aid in determining the coordinates. In another embodiment, the variables may further include the length of the mouse's major and minor axes, and the sine and cosine of the vector angle of the major axis. This angle may be defined relative to the direction of the major axis. The major axis may extend from about the tip of the animal's head (e.g., nose) to the end of the animal's body (e.g., about the point where the animal's tail extends from the body) in the coordinate system of the video frame for clarity, while cropped frames are shown herein as input to the neural network, the actual input is a full, unmarked frame.

[0146]

[0199] The neural network architecture is used to determine the ellipse parameters. Exemplary systems and methods for this are discussed in detail below. It will be appreciated that other parameters may be utilized and determined in accordance with the disclosed embodiments, as desired.

[0147]

[0200] In one embodiment, the first architecture is an encoder-decoder segmentation. As shown in Figure 9, this network predicts a foreground-background segmented image from a given input frame and outputs a segmentation mask that can predict whether a mouse is present in pixel terms.

[0148]

[0201] This first architecture splits the input into a set of small spatial resolution features (e.g. The model has a feature encoder configured to abstract the image to a matrix (e.g., 5x5 for 480x480). A number of parameters are assigned to the neural network for learning. Learning can be performed by supervised training, where the neural network is presented with examples and parameters are adjusted to produce correct predictions. The final model definition and training hyperparameters are all listed in Table 3 below.

[0149] [Table 3]

[0150]

[0202] The feature encoder encodes a set of small spatial resolution features into a matrix with the same shape as the original input image. This is followed by a feature decoder configured to restore the original state, i.e., the learned parameters in the neural network reverse the feature encoding operation.

[0151]

[0203] Three fully connected layers are added to the encoded features to predict the cardinal directions the ellipses face. A fully connected layer may represent a neural network layer in which each digit in a given layer is multiplied by a different parameter (e.g., a learnable parameter) and the sum produces a single value in the new layer. This feature decoder can be trained to generate a foreground-background segmented image.

[0152]

[0204] The first half of the network (encoder) uses batch normalization and ReLu activation. It utilizes a 2D convolutional layer followed by a 2D max-pooling layer. Further details can be found in Goodfellow (2016).

[0153]

[0205] A starting filter size of 8 was adopted, which doubled after each pooling layer. The kernels used are of shape 5x5 for the 2D convolutional layers and 2x2 for the max-pooling layers. The input video is of shape 480x480x1 (e.g., monochrome), and after six iterations through these layers, the resulting shape is 15x15x128 (e.g., 128 colors).

[0154]

[0206] In alternative embodiments, pooling layers of other shapes, such as 3x3, are employed. Recursive layers represent layers with a recursive structure. The neural network learns different parameters for each layer, and each layer is stacked. Six recursive layers are shown above, but more or fewer recursive layers can be used.

[0155]

[0207] Another 2D convolution layer (kernel 5x5, 2x filter) is applied, 2D max pooling is applied with a different kernel 3x3 and stride 3. The 15x15 spatial shape can be further reduced by using a factor of 3. Normal max pooling is kernel 2x2, stride 2, but each 2x2 grid selects the maximum and generates one value. These settings select the maximum value in a 3x3 grid.

[0156]

[0208] A final 2D convolutional layer is applied to generate a 5x5x512 shape feature bottle. A bottleneck is generated. The feature bottleneck represents the encoding feature set, and the actual matrix values ​​output by all these matrix operations. The learning algorithm optimizes the encoding feature set to be most meaningful for the task it is trained to perform well. This feature bottleneck is then passed to both the segmentation decoder and the angle predictor.

[0157]

[0209] The segmentation decoder uses strided transposed 2D convolutional layers. The encoder is reversed and the pre-downsampling activations are carried forward by summation junctions. Note that this decoder does not utilize ReLu activations. The pre-downsampling activations and summation junctions can also be referred to as skip connections. Since the feature in the layer where the decoding matches the same shape as the encoder layer, the network can choose between better encoding and keeping the state at the encoder state.

[0158]

[0210] After the layer returns to a 480x480x8 shape, another kernel size of 1x1 is used. Applying convolutions results in two monochrome images with depth (a background prediction and a foreground prediction). The final output is 480x480x2 (two colors). The first color is designated to represent the background. The second color is designated to represent the foreground. For each pixel, the network considers the larger of the two as the input pixel. As discussed below, a softmax operation rescales these colors so that their cumulative probabilities sum to one.

[0159]

[0211] Then softmax is applied over this depth. It is a form of classification into groups or binmin. Further information on softmax can be found in Goodfellow (2016).

[0160]

[0212] The feature bottleneck also generates an angle prediction, which is a function of two 2D convolutions. This is achieved by applying batch normalization and ReLu activation to the flattened layer (kernel size 5x5, feature depths 128 and 64). From here, one fully connected layer is flattened and used to generate a 4-neuron shape that acts to predict the quadrant the mouse's head will face. Further details on batch normalization, ReLu activation, and flattening can be found in Goodfellow (2016).

[0161]

[0213] The angle is predicted by the segmentation mask, so the correct direction (± Only one direction (+180°) needs to be chosen. That is, because an ellipse is predicted, there is only one major axis. One end of the major axis is the direction of the mouse's head. The mouse is assumed to be longer along the head-to-tail axis. Thus, one direction is +180° (head) and the other is -180° (tail). The four possible directions that the encoder-decoder neural network architecture can choose from are 45 to 135°, 135 to 225°, 225 to 315°, and 315 to 45° on a polar grid.

[0162]

[0214] These boundaries were chosen to avoid discontinuities in the angle prediction. In particular, as mentioned above, the angle prediction is a prediction of the sine and cosine of the vector angle of the major axis, employing the atan2 function. The atan2 function has discontinuities (at 180°), and the selected boundaries avoid these discontinuities.

[0163]

[0215] After the network generates the segmentation mask, Branson ( An ellipse fitting algorithm can be applied for tracking, as described in Branson (2009). Branson uses weighted sample means and variances in these calculations, but the segmentation neural network remains invariant to the conditions that represent improvements. For segmentation masks generated by background subtraction algorithms, projected shadows can add errors. The neural network learns to avoid these issues. Furthermore, no significant differences are observed between using weighted and unweighted sample means and variances. Ellipse fitting parameters predicted by the weighted and unweighted approaches do not differ significantly using masks predicted by the disclosed neural network embodiments.

[0164]

[0216] Given a segmentation mask, the sample mean of the pixel location is the center position is calculated to represent

[0165]

number

[0166] Similarly, the sample variance of the pixel location is calculated to represent the major axis length (a), the minor axis length (b), and the angle (θ).

[0167]

number

[0168] To find the axis length and angle, an eigenvalue decomposition equation must be solved.

[0169]

number

[0170]

number

[0171]

[0217] The second network architecture is a binning classification network. As shown in Figure 10, the structure of the binning classification network architecture is able to predict a heat map of the most probable values ​​for each ellipse fitting parameter.

[0172]

[0218] This network architecture abstracts the input image to a smaller spatial resolution. We start with a feature encoder that encodes the solution. While most regression predictors implement the solution via a bounding box (e.g., a square or rectangle), the ellipse only adds one additional parameter, the angle. Because this is the iteration number where the angle is equal to 360° and 0°, the angle parameter is converted to its sine and cosine components. This results in a total of six parameters regressed from the network. The first half of this network encodes a set of features relevant to solving the problem.

[0173]

[0219] The encoded features convert the feature matrix (array) into a single vector. The flattened encoded features are then fed into additional fully connected layers (e.g., by feeding the feature vector into a fully connected layer) where the output shape is determined by the desired output resolution. For example, for the x-coordinate position of the mouse, there are 480 bins, one for each x-column of a 480x480 pixel image.

[0174]

[0220] When the network runs, the maximum value in each heatmap is selected as the most probable value. Each desired output parameter can be realized as a set of independent trainable fully connected layers connected to the coding features.

[0175]

[0221] Resnet V2 50, Resnet V2 101, Resnet V A wide variety of pre-built feature detectors were tested, including Resnet V2 200, Inception V3, Inception V4, VGG, and Alexnet. Feature detectors represent convolutions that operate on the input image. In addition to these pre-built feature detectors, a wide variety of custom networks were also investigated. From this investigation, Resnet V2 200 was observed to perform best.

[0176]

[0222] The final architecture is the recurrent network shown in Figure 11. The regression network then takes an input video frame, extracts features using the Resnet200 CNN, and directly predicts six parameters for ellipse fitting. Each value (six for ellipse fitting) is continuous and can have an infinite range. The network needs to learn the appropriate range of values. In this way, the numerical values ​​of the ellipse describing the tracking ellipse are predicted directly from the input image. That is, instead of directly predicting the parameters, the regression network instead selects the most probable value from a selection of possible binning values.

[0177]

[0223] Other neural network architectures work differently. The encoder-decoder neural network architecture outputs the probability that each pixel is a mouse or not. The binning classification neural network architecture outputs a bin that represents the mouse's location. The class of each parameter is predetermined, and the network (encoder-decoder or binning) only needs to output the probability of each class.

[0178]

[0224] Recurrent network architectures abstract inputs to a finer spatial resolution. In contrast to the architectures above, regression neural network training relies on a cross-entropy loss function, as opposed to a mean squared error loss function.

[0179]

[0225] Due to memory constraints, we reduced the feature dimensionality and used only a custom VGG-like network. were tested. The best performing network was structured with two 2D convolutional layers followed by a 2D max-pooling layer. The kernels used were 3x3 in the 2D convolutional layers and 2x2 in the 2D max-pooling layers. The initial filter depth used was 16, which was doubled for each 2D max-pooling layer. This two-convolution + max-pooling sequence was repeated five times, resulting in a 15x15x256 geometry.

[0180]

[0226] This layer is flattened and connected to a fully connected layer for each output. The shape is determined by the desired resolution and range of prediction. As an example, these encoded features were then flattened and connected to a fully connected layer, resulting in an output shape of 6, which is the number of values ​​the network was required to predict fit to an ellipse. For testing purposes, only center positions were observed and the network was trained on a wide range of global images (0-480). Additional outputs, such as angle predictions, can easily be added as additional output vectors. While a variety of contemporary feature encoders were tested, the data discussed here for this network comes from the 200-layer Resnet V2 (He (2016)), which achieved the best-performing results for this architecture.

[0181] Training dataset

[0227] To test the network architecture, we used OpenCV, as described below. A training dataset consisting of 16,234 training images and 568 distinct validation images across multiple strains and environments was generated using a based labeling interface that allows fast foreground and background labeling as well as ellipse fitting, and can be used to immediately generate training data and adapt any network to new experimental conditions through transfer learning.

[0182]

[0228] Interactive watershed-based segmentation The OpenCV library was employed to generate the contour- and contour-based ellipse fitting. Using this software, the user marks points as foreground (e.g., mouse (F)) by left-clicking and labels other points as background (B) by right-clicking, as shown in Figure 12A. A keystroke executes the watershed algorithm, which predicts the segmentation and ellipse, as shown in Figure 12B. If the user needs to edit the predicted segmentation and ellipse, they simply label more areas and run the watershed again.

[0183]

[0229] Any selection made by the user of the neural network (e.g., a researcher) If the prediction falls within a predetermined error tolerance, the user selects the orientation of the ellipse. The user makes the selection by choosing one of four cardinal directions (up, down, left, or right). The ellipse fitting algorithm selects the exact angle, so the user only needs to identify ±90° of the orientation. Once the orientation is selected, all relevant data is saved and the user is presented with a new frame to label.

[0184]

[0230] The goal of the labeled dataset is to obtain good ellipse-fitting tracking for mice. The goal of this study was to identify the data. During data labeling, ellipse fitting was optimized so that the center of the ellipse was the body of the mouse, with the end of the long axis approximately touching the nose of the mouse. The tail was often removed from the segmentation mask to provide a better ellipse fit.

[0185]

[0231] To train the network for inference, we use three labeled training sets. Each training set included a reference frame (input), a segmentation mask, and an ellipse fitting. Each training set was generated to track a mouse in a different environment.

[0186]

[0232] The first environment is an audiovisual environment with a constant white background containing 16,802 annotated frames. The first 16,000 frames were labeled from 65 separate videos taken from one of 24 identical setups. After the first training run of the network, it was observed that the network performed poorly in specific situations not included in the labeled data. Instances of mid-arena jumping, irregular postures, and urination were typically observed as failures. These failures were identified, correctly labeled, and incorporated into the labeled training set to further generalize and improve performance.

[0187]

[0233] The second environment was an ALPHA-dri bedding and feeding area with two different The dataset consisted of a standard open field under different lighting conditions (visible light during the day and infrared light at night). A total of 2,192 frames were labeled across six setups over four days. 916 of the annotated frames were obtained under nighttime illumination, and 1,276 of the annotated frames were obtained under daytime illumination.

[0188]

[0234] The final labeled dataset is the Opto- Generated using an M4 open-field cage, this dataset contained 1,083 labeled frames, all sampled across different videos (one frame per video) and eight different setups.

[0189] Neural Network Training a) Expanding the training dataset

[0235] This training dataset is trained by applying reflections. During training, the network was expanded by a factor of eight, applying small random variations in contrast, brightness, and rotation to make the network robust to small variations in the input data. This expansion is performed to prevent the neural network from memorizing the training dataset, which would cause it to perform poorly on examples not included in the dataset (validation). Further details can be found in Krizhevsky (2012).

[0190]

[0236] Training set expansion has been a key feature of neural networks since Alexnet. This is an important aspect of training (Krizhevsky (2012)). A handful of training set augmentations are utilized to achieve good regularization performance. Because the data comes from a bird's-eye view, it is easy to apply horizontal, vertical, and oblique reflections to quickly increase the training set size by 8x for a comparable training set size. Also, slight rotations and translations are applied to the entire frame at runtime. The rotation augmentation values ​​are sampled from a uniform distribution. Finally, noise, brightness, and contrast augmentations can also be applied to the frames. The random values ​​used for these augmentations are chosen from normal distributions.

[0191] b) Learning rate and batch size for training

[0237] The learning rate and batch size of the training are Large networks such as Resnet V2 200 can run into batch size memory constraints with an input size of 480x480, but grid search is not possible. Using a search technique, good learning rates and batch sizes were identified experimentally. The hyperparameters selected for training these networks are shown in Table 3 above.

[0192] Model

[0238] In Tensorflow v1.0, model building, training, and The presented training benchmarks were run on an NVIDIA® Tesla® P100 GPU architecture.

[0193]

[0239] The hyperparameters are trained through multiple training iterations. After the first training of the network, it was observed that the network performed poorly in special situations that were underrepresented in the training data. Instances of mid-arena jumping, irregular postures, and urination were typically observed as failures. These challenging frames were identified and incorporated into the training dataset to further improve performance. A complete description of the final model definition and all training hyperparameters are listed above in Table 3.

[0194]

[0240] The training and validation loss curves shown by all three networks The training and validation loss curves are shown in Figures 13A–13E, respectively. Overall, the training and validation loss curves show that all three networks were trained to an average error of 1–2 pixels. Unexpectedly, the binned classification network exhibited unstable loss curves, indicating overfitting and poor generalization during validation (Figures 13B–13E). The regression architecture converged to a validation error of 1.2 pixels, indicating better training performance than validation (Figures 13A–13D). However, the best-performing feature extractor, Resnet V2 200, is a large deep network with over 200 layers and 62.7 million parameters, resulting in substantially longer processing times per frame (33.6 ms). Other pre-built general-purpose networks (Zoph (2017)) achieve similar or lower performance, at the expense of shorter computation times. Thus, regression networks are an accurate but computationally expensive solution.

[0195]

[0241] As further shown in Figures 13A, 13B, and 13C, The DA segmentation architecture converged to a verification error of 0.9 pixels. Not only did the segmentation architecture perform well, but it was also computationally efficient on GPU operations, with an average processing time of 5-6 ms per frame. Video data could be processed at up to 200 fps (6x real-time) on a server-level GPU, the Nvidia Tesla P100, and at 125 fps (4.2x real-time) on a consumer-level GPU, the Nvidia Titan Xp. This high processing speed is likely due to the structure's depth of only 18 layers and 10.6 million parameters.

[0196]

[0242] Encoder-Decoder Segmentation Network Architecture A benchmark of training set size was also conducted to identify the relative scale of labeled training data required for good network performance. This benchmark was tested by shuffling and randomly sampling subsets of the training set (e.g., 10,000, 5,000, 2,500, 1,000, and 500). Each subsampled training set was trained and compared to an identical validation set. The results of this benchmark are shown in Figures 14A-14H.

[0197]

[0243] In general, the training curves appear indistinguishable (Fig. 14A). The training set size shows no change in performance in terms of training set error rate (Figure 14A). Surprisingly, while validation performance converges to the same value above 2,500 training samples, the error increases below 1,000 training samples (Figure 14B). As further illustrated, validation accuracy exceeds training accuracy above 2,500 training samples (Figures 14C-14F), but begins to show signs of weak generalization after matching training accuracy at 1,000 (Figure 14G). Using only 500 training samples clearly results in overfitting, as indicated by the divergence and increasing validation error rate (Figure 14H). This suggests that the training set is no longer large enough for the network to generalize well. Therefore, good results are obtained from networks trained with only 2,500 labeled images, which takes approximately 3 hours to generate using the labeling interface. Therefore, while the exact number of training samples will ultimately depend on the difficulty of the visual problem, a recommended starting point is around 2,500 training samples.

[0198]

[0244] An exemplary video frame showing a mouse being tracked in accordance with the disclosed embodiments is: 15A and 15B under visible light and 15C and 15D under infrared light, respectively. As shown, the spatial extent of each individual mouse is color-coded on a pixel-by-pixel basis.

[0199]

[0245] Computational efficiency, accuracy, training stability, and small training data requirements Given the data, an encoder-decoder segmentation architecture was chosen for predicting the mouse position throughout the video for comparison with other methods.

[0200]

[0246] Inferring entire videos from mice with different coat colors and data collection environments (Figure 8A) The quality of the neural network-based tracking was evaluated by visually assessing the quality of the tracking. The neural network-based tracking was also compared with an independent tracking modality, the KOMP2 beam break system (Figure 8A, column 6).

[0201] Experimental Arena a) Open Field Arena

[0247] One embodiment of arena 200 is employed as an open field arena. The open field arenas were 52 cm x 52 cm. The floor was white PVC plastic and the walls were gray PVC plastic. A white 2.54 cm surface was added to all inner edges to aid in cleaning and maintenance. Illumination was provided by an LED light ring (model F&V R300). The light ring was calibrated to produce 600 lux of light in each arena.

[0202] b) Open-field arena with 24-hour monitoring

[0248] The open field arena was expanded for several days of testing. The lighting was in the form of overhead LED lighting set to a standard 12:12 LD cycle. Alpha Dry was placed in the arena as bedding. A single Diet Gel 76A food trough was placed in the arena to provide food and water. This nutrient source was monitored and replaced when depleted. Each matrix was illuminated at 250 lux during the day and approximately <500 lux at night. For nighttime video recording, lighting 212 included IR LED (940 nm) illumination.

[0203] c) KOMP Open Field Arena

[0249] In addition to custom arenas, embodiments of the disclosed systems and methods may be used with commercially available arenas. A benchmark was also performed on the stem. The Opto-M4 open-field cage was constructed using transparent plastic walls. This made visual tracking very difficult due to the resulting reflections. The cage measured 42 cm x 42 cm. The arena was illuminated with LEDs at 100-200 lux.

[0204] Video Acquisition

[0250] All video data was acquired using the video acquisition system discussed with respect to Figures 2 and 7. Video data was acquired using an embodiment of the system. Video data was acquired using a camera 210 in the form of a Sentech camera (model STC-MB33USB) and a computer lens (model T3Z2910CS-IR) at a resolution of 640 x 480 pixels, 8-bit monochrome depth, and approximately 29 fps (e.g., approximately 29.9 fps). Exposure time and gain were digitally controlled using a target brightness of 190 / 255. The aperture was adjusted to its widest setting so that low analog gain was used to achieve the target brightness, thereby reducing the amplification of baseline noise. Files were temporarily saved to a local hard drive using the "raw video" codec and "pal8" pixel format. The assay ran for approximately 2 hours and generated approximately 50 GB of raw video files. The ffmpeg software was used overnight to compress using the MPEG4 codec (quality set to maximum) which applied a noise reduction filter with a 480x480 pixel crop and produced a compressed video size of approximately 600MB.

[0205]

[0251] In order to reduce the projection distortion, the frame 202 is positioned approximately 100 cm above the shelf 202b. A camera 210 was mounted on the KOMP arena. Zoom and focus were manually set to achieve a zoom of 8 pixels / cm. This resolution minimizes unused pixels on the arena borders and produces an area of ​​approximately 800 pixels per mouse. The KOMP arena is slightly smaller, but utilized the same target zoom of 8 pixels / cm.

[0206]

[0252] Encoder-Decoder Segmentation Using Neural Networks Using this method, 2002 videos (700 hours total) from the KOMP2 dataset were tracked, and the results are shown in Figure 8. These data included 232 knockout lines tested in a 20-minute open-field assay on a C57BL / 6NJ background. Because each KOMP2 arena had a slightly different background due to the transparent matrix, tracking performance was compared for each of the eight test chambers (n = 250 on average (Figure 16)) and for all combination boxes. Across all eight test chambers used by KOMP2, a very high correlation was observed between the total distance traveled in the open field between the two methods (R = 96.9%). This trend (red arrows) indicates that two animals were observed with high discrepancies. Video observations indicated an irregular posture present in both animals, with one step being a waddling gait and the other a hunched posture. Waddling and hunched gaits likely resulted in abnormal beam breaks, resulting in abnormally high total distance traveled measures from the beam-break system. This example highlights one of the advantages of neural networks: their insensitivity to the animal's posture.

[0207]

[0253] Regarding the performance of the trained segmentation neural network, Even though the BGS library was used, it was compared with Ctrax across a wide range of videos from various test environments and coat colors, as described above with respect to Figure 8A. The comparison with Ctrax was motivated by a number of reasons. On the one hand, Ctrax is considered one of the best trackers to date, allowing fine-tuning of many tracking settings. Furthermore, Ctrax is open source and provides user support. Given the results with the BGS library (Figure 8B), similar or inferior performance is expected for other trackers. Twelve animals per group were tracked with both the trained segmentation neural network and Ctrax. Ctrax settings were fine-tuned for each of the 72 videos, as described below.

[0208]

[0254] Ctrax includes a variety of settings to optimize tracking performance (Branso n (2009)). The authors of this software strongly recommend that the arena be set up under specific criteria to ensure good tracking. Most of the tests discussed herein (e.g., albino mice on a white background) employ environments in which Ctrax is not designed to function well. Nevertheless, good performance can still be achieved if parameters are properly adjusted. Due to the many settings required for operation, Ctrax can easily become time-consuming to achieve good tracking performance. The procedure for setting up Ctrax to track mice in the disclosed environment is as follows.

[0209]

[0255] In the first operation, a background model is generated. Because it is based on subtraction, having a robust background model is essential for its functionality. The model works best when the mouse is moving. To generate the background model, it finds parts of the video where the mouse is clearly moving and samples frames from those parts. This ensures that the mouse is not included in the background model. This approach significantly improves Ctrax's tracking performance on 24-hour data, as the mouse does not move much and is usually incorporated into the background model.

[0210]

[0256] The second action is to set the background subtraction, where the standard range is 254 A background luminance normalization method ranging from 0.9 to 255.0 is used. The thresholds applied to separate mice are adjusted based on preliminary video, as slight variations in exposure and coat color can affect performance. To adjust these thresholds, a set of good starting values ​​is applied, and the video is scrutinized to ensure generally good performance. In certain embodiments, all videos may be checked for cases of mice with their backs to a wall, as these are typically the most difficult frames to track due to shadows. Morphological filtering may also be applied to remove subtle changes in the environment and to remove the mouse's tail for ellipse fitting. An aperture radius of 4 and an occlusion radius of 5 were adopted.

[0211]

[0257] In another operation, Ctrax allows you to Various tracking parameters were manually adjusted. To accommodate time constraints, these parameters were fully adjusted before and after use for all other tracked mice. When a video clearly performed poorly, general settings were fine-tuned to improve performance. For shape parameters, a range based on two standard deviations was determined from individual black mouse videos. Anticipating that certain mice would perform poorly in the segmentation step, the minimum value was further lowered. This allowed Ctrax to still find a good mouse location, even though it was unable to segment the entire mouse. This method performed well because all setups had the same zoom of 8 and the mice tested had roughly the same shape. In our experimental setup, we tracked only one mouse in the arena, so the motion settings were very lenient. Under the observation parameters, we primarily used "Min Area Ignore," which removes large detections. Detections larger than 2,500 were removed. Under the Hindsight tab, the "Fix Spurious Detections" setting is used to remove detections shorter than 500 frames in length.

[0212]

[0258] The inability of Ctrax to generate a valid background model means that animals are not captured for long periods of time. Videos from the 24-hour device were manually omitted from the comparison. The cumulative relative error in total distance traveled between Ctrax and the neural network was calculated and is shown in (Figure 17A). For every minute of the video, the distance traveled predictions from both the neural network and Ctrax are compared. This metric measures the accuracy of center of mass tracking for each mouse. Black, gray, Tracking of the albino and spotted mice showed less than 4% error, although significantly higher levels of error were observed in the albino (14%), 24-hour arena (27% (orange)), and KOMP2 (10% (blue)) (Figure 17A). Thus, without the neural network tracker, the albino, KOMP2, or 24-hour data could not be properly tracked.

[0213]

[0259] In addition, the foreground segmentation prediction may be incorrect when shadows are included in the prediction. In these cases, even if center of mass tracking was possible, the ellipse fitting itself had high variability.

[0214]

[0260] Modern machine learning for action recognition, such as JAABA (Kabra (2013)). The software utilizes these features for behavioral classification. The variance in ellipse tracking is quantified by the relative standard deviation of the minor axis and is shown in Figure 17B. This metric exhibits the least variance across all experimental mice, as the width of individual mice remains similar across the wide range of postures represented in behavioral assays when tracking is accurate. High tracking variance was observed for gray and mottled mice (Figure 17A), even though the cumulative relative error in total distance traveled was small (Figure 17B). As expected, high relative standard deviations are observed for the minor axis in albino and KOMP2 tracking. This demonstrates that the neural network tracker outperforms conventional trackers in both centroid tracking and ellipse fitting variance.

[0215]

[0261] High-precision encoder-decoder segmentation neural network Having constructed a temperature tracker, its performance was further tested with two large behavioral datasets. Open-field video data were generated for 1,845 mice (1,691 hours) across 58 mouse strains, including all colored, spotted, nude, and obese mice. This dataset includes 47 inbred and 11 F1 isogenic mouse strains and is the largest open-field dataset generated according to the Mouse Phenome Database by Bogue (2018).

[0216]

[0262] The tracking results for total distance traveled are shown in Figure 18A. Each point represents an individual in the lineage. Boxes indicate mean ± standard deviation. All mice were tracked with high accuracy using a single trained network without user adjustments. Tracking fidelity was visually confirmed for the majority of mouse strains, and excellent performance was observed. The observed motor phenotype is consistent with published datasets of mouse open-field behavior.

[0217]

[0263] Four C57BL / 6J mice and two BTBR T + ltpr3 tf The same neural network was employed to track 24-hour video data collected on 1 / J mice (Figure 8A, column 5). These mice were housed with bedding, food, and water bowls for several days, with food repositioning and a 12:12 light / dark lighting regime. Video data was recorded using visible and infrared light sources. Using the same network under these conditions, the movements of all animals were tracked, and excellent performance was observed under both light and dark conditions.

[0218]

[0264] The results are shown in Figure 18B, where eight bright and dark points are observed under light and dark conditions, respectively. As expected, a locomotor rhythm (curve) accompanied by a high level of locomotor activity was observed during the dark period.

[0219]

[0265] In summary, video-based tracking of animals in complex environments is a promising tool for understanding animal behavior. This has been a long-standing challenge in the field (Egnor (2016)). Current state-of-the-art systems do not address the fundamental problem of animal segmentation and rely heavily on visual contrast between the foreground and background for accurate tracking. As a result, users must constrain their environment to achieve optimal results.

[0220]

[0266] Herein, we focus on modern neural networks that can function in complex and dynamic environments. A neural network-based tracker and corresponding methods of use are described. A fundamental problem in tracking (foreground and background segmentation) is addressed through the use of trainable neural networks. Testing three different architectures shows that an encoder-decoder segmentation network achieves a high level of accuracy and operates at high speed (more than six times faster than real time).

[0221]

[0267] By labeling only 2,500 images (approximately 3 hours), In addition, a labeling interface is provided that allows new networks to be trained for specific environments.

[0222]

[0268] The disclosed trained neural network outperforms two existing solutions. The proposed method was compared to other methods and found to significantly outperform them in complex environments. Similar results would be expected for any commercially available system that utilizes background subtraction techniques. Indeed, when 26 different background subtraction methods were tested, each was observed to be unsuccessful under certain circumstances. However, a single neural network architecture could function for mice of all coat colors in multiple environments without the need for fine-tuning or user input. This machine learning approach forms the basis of a next-generation tracking architecture for behavioral research, as it enables long-term tracking under dynamic environmental conditions with minimal user input.

[0223]

[0269] One or more aspects or features of the control systems described herein may include digital The present invention may be implemented in digital electronic circuitry, integrated circuits, specially designed application-specific integrated circuits (ASICs), field programmable gate array (FPGA) computer hardware, firmware, software, and / or combinations thereof. Various aspects or features may include implementation in one or more computer programs executed and / or interpreted on a programmable system including at least one programmable processor, whether special or general purpose, which may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Programmable or computer systems include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0224]

[0270] Programs, software, software applications, applications A computer program, which may also be referred to as an application, component, or code, includes machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, and / or an assembly / machine language. As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device, such as, for example, magnetic disks, optical disks, memory, and programmable logic devices (PLDs), used to provide machine instructions and / or data to a programmable processor (including a machine-readable medium that receives machine instructions as a machine-readable signal). The term "machine-readable signal" refers to any computer program product, apparatus, and / or device, such as, for example, magnetic disks, optical disks, memory, and programmable logic devices (PLDs), used to provide machine instructions and / or data to a programmable processor (including a machine-readable medium that receives machine instructions as a machine-readable signal). "nal") refers to any signal used to provide machine instructions and / or data to a programmable processor. The machine-readable medium may persistently store such machine instructions, such as, for example, a non-transitory solid-state memory, a magnetic hard drive, or any equivalent storage medium. Alternatively or additionally, the machine-readable medium may persistently store such machine instructions, such as, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0225]

[0271] A cathode to allow interaction with the user, e.g., to display information to the user One or more aspects or features of the subject matter described herein may be implemented on a computer having a display device, such as a cathode ray tube (CRT), liquid crystal display (LCD), or light emitting diode (LED) monitor, as well as a keyboard and pointing device (e.g., a mouse, trackball, etc.) through which a user can provide input to the computer. Other types of devices that allow for user interaction may be used as well. Feedback provided to the user can be any form of sensory feedback, for example, visual, auditory, or tactile feedback, and input from the user can be accepted in any form, including, but not limited to, acoustic, speech, or tactile input. Other possible input devices include, but are not limited to, touchscreens or other touch-sensitive devices such as single-point or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, etc.

[0226]

[0272] All references (e.g., published or registered) cited throughout this application All references (including, but not limited to, patents or equivalents, published patent applications, and non-patent or other source materials) are incorporated herein by reference in their entirety, as if each reference were individually incorporated by reference, to the extent that each reference is at least partially consistent with the disclosure of this application. For example, a partially conflicting reference is incorporated by reference except for the partially conflicting portion.

[0227]

[0273] In this specification, when a Markush group or other group is used, the group It is intended that all individual elements of the groups and all possible combinations and subcombinations of the groups are individually included in the disclosure.

[0228]

[0274] As used herein, the singular forms "a," "an," and " The word "the" includes the plural unless the context clearly dictates otherwise. Thus, for example, reference to "a cell" includes a plurality of such cells and equivalents thereof known to those skilled in the art, and so forth. Additionally, the terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably herein.

[0229]

[0275] In this specification, the term "comprising" means "comprising" The terms "including," "having," "containing," and "characterized by" are synonymous with, and may be used interchangeably with, each of these terms. Each of these terms is also inclusive or open-ended and does not exclude additional, unrecited elements or method steps.

[0230]

[0276] As used herein, the term "consisting of" means Any element, step, or ingredient not specified in a claim element is excluded.

[0231]

[0277] As used herein, the term "consisting essentially of" The term "comprising," "consisting essentially of," and "consisting of" does not exclude elements or steps that do not materially affect the basic and novel characteristics of the claim. Wherever used in this specification, any of the terms "comprising," "consisting essentially of," and "consisting of" may be replaced by either of the other two terms.

[0232]

[0278] The embodiments illustratively described herein are not intended to be limiting unless specifically disclosed herein. The present invention may be suitably practiced in the absence of any one or more of the elements or limitations not listed above.

[0233]

[0279] The expression "(of any of claims XX to YY)" The phrase "as in any one of claims XX-YY") (XX and YY represent claim numbers) is intended to provide an alternative form of multiple dependent claims and in some embodiments may be used interchangeably with the phrase "as in any one of claims XX-YY."

[0234]

[0280] Unless otherwise specified, all technical terms and Scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosed embodiments belong.

[0235]

[0281] As used herein, for example, a temperature range, a time range, a composition range, or a concentration range Whenever a range such as 100 is given, it is intended that all intermediate ranges and subranges, as well as all individual values ​​within the given range, are included in the disclosure. Ranges herein specifically include the values ​​provided as the endpoints of the range. For example, a range of 1 to 100 specifically includes the endpoints 1 and 100. It is understood that any subranges included herein, as well as individual values ​​within a range or subrange, may be excluded from the claims.

[0236]

[0282] In the above and in the claims, "at least one of Expressions such as "at least one of" or "one or more of" may appear followed by a conjunctive list of elements or features. Also, the term "and / or" may appear as a list of two or more elements or features. Unless the context of use indicates otherwise, implicitly or explicitly, such expressions are intended to mean any of the elements or features of the list individually, or any of the listed elements or features in combination with any of the other listed elements or features. For example, the expressions "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or A and B in combination," respectively. A similar interpretation is intended for lists containing more than two items. For example, the expression "at least one of A, B, and C" The terms "one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, the combination of A and B, the combination of A and C, the combination of B and C, or the combination of A, B, and C," respectively. Also, above and in the claims, use of the term "based on" is intended to mean "based at least in part on," so that unrecited features or elements are allowed.

[0237]

[0283] The terms and expressions employed in this specification are used as terms of description, No limitation is intended, and the use of such terms and expressions is not intended to exclude any equivalents of the features shown and described or portions thereof, while recognizing that various modifications are possible within the scope of the claimed embodiments. Thus, while this application may include descriptions of preferred embodiments, exemplary embodiments, and optional features, it is understood that modifications and variations of the concepts disclosed herein may be made by those skilled in the art. Such modifications and variations are considered to be within the scope of the disclosed embodiments, as defined by the appended claims. The specific embodiments described herein are examples of useful embodiments of the disclosure, which will be apparent to those skilled in the art and may be implemented using many variations of the devices, device components, and method steps described herein. As will be apparent to those skilled in the art, the methods and devices useful for the methods may include many optional configurations, processing elements, and steps.

[0238]

[0284] The embodiments of the present disclosure may be modified in other ways without departing from the spirit or essential characteristics thereof. The foregoing embodiments, therefore, are to be considered in all respects illustrative and not restrictive of the subject matter described herein.

[0239] References

[0285] Each of the references listed below is incorporated herein by reference in its entirety. be incorporated into the book.

[0240] [ka]

[0241] [ka]

[0242] [ka]

Claims

1. 1. A method for tracking animals, comprising: receiving, by a processor, video data representing an observation of an animal; The processor: receiving input video frames extracted from the video data; generating an ellipse description of at least one animal based on the input video frames, the ellipse description being defined by predetermined ellipse parameters; and providing data for said at least one animal including values ​​characterizing said predetermined ellipse parameters; executing a neural network architecture configured to: A method comprising:

2. 2. The method of claim 1, wherein the ellipse parameters are coordinates representing the position of the animal in a plane, the length of the animal's major axis and the length of the animal's minor axis, and the angle at which the animal's head is pointing, defined relative to the direction of the major axis.

3. said neural network architecture comprising: predicting a foreground-background segmented image from an input video frame; predicting, in terms of pixels, whether an animal is present in the input video frame based on the segmented image; outputting a segmentation mask based on the pixel-wise prediction; and fitting an ellipse to the portion of the segmentation mask where the animal is predicted to be present to determine values ​​characterizing the predetermined ellipse parameters; 2. The method of claim 1, wherein the network is an encoder-decoder segmentation network configured to:

4. the encoder-decoder segmentation network: a feature encoder configured to abstract the input video frame into a set of small spatial resolution features; a feature decoder configured to transform the set of features into the same shape as the input video frame and output the foreground-background segmented image; an angle predictor configured to predict an angle at which the animal's head is pointing; The method of claim 3 comprising:

5. The method of claim 1 , wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of most probable values ​​for each ellipse parameter of the ellipse description.

6. 6. The method of claim 5, wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, the abstraction being utilized to generate the heatmap.

7. The method of claim 1 , wherein the neural network architecture comprises a recurrent network configured to extract features from input video frames and directly predict values ​​characterizing each of the ellipse parameters.

8. The method of claim 1 , wherein the animal is a rodent.

9. 1. A system for tracking animals, comprising: a data storage device for maintaining video data representing observations of the animals; a processor configured to receive video data from the data storage device and to implement a neural network architecture, the neural network architecture comprising: receiving input video frames extracted from the video data; generating an ellipse description of at least one animal based on the video frames, the ellipse description being defined by predetermined ellipse parameters; and providing data for said at least one animal including values ​​characterizing said predetermined ellipse parameters; A system configured to:

10. 10. The system of claim 9, wherein the ellipse parameters are coordinates representing the animal's position within a plane, the length of the animal's major axis and the length of the animal's minor axis, and the angle at which the animal's head is pointing, defined relative to the direction of the major axis.

11. said neural network architecture comprising: predicting a foreground-background segmented image from an input video frame; predicting, in terms of pixels, whether an animal is present in the input video frame based on the segmented image; outputting a segmentation mask based on the pixel-wise prediction; and fitting an ellipse to the portion of the segmentation mask where the animal is predicted to be present to determine values ​​characterizing the predetermined ellipse parameters; 10. The system of claim 9, wherein the network is an encoder-decoder segmentation network configured to:

12. the encoder-decoder segmentation network: a feature encoder configured to abstract the input video frame into a set of small spatial resolution features; a feature decoder configured to transform the set of features into the same shape as the input video frame and output the foreground-background segmented image; an angle predictor configured to predict an angle at which the animal's head is pointing; The system of claim 11 , comprising:

13. 10. The system of claim 9, wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of most probable values ​​for each ellipse parameter of the ellipse description.

14. 14. The system of claim 13, wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, the abstraction being utilized to generate the heatmap.

15. 10. The system of claim 9, wherein the neural network architecture comprises a recurrent network configured to extract features from input video frames and directly predict values ​​characterizing each of the ellipse parameters.

16. The system of claim 9 , wherein the animal is a rodent.

17. 1. A non-transitory computer program product storing instructions that, when executed by at least one data processor of at least one computing system, receiving video data representing an observation of an animal; Executing a neural network architecture; and performing a method comprising: receiving input video frames extracted from the video data; generating an ellipse description of at least one animal based on the input video frames, the ellipse description being defined by predetermined ellipse parameters; and providing data for said at least one animal including values ​​characterizing said predetermined ellipse parameters; 1. A non-transitory computer program product configured to:

18. 18. The computer program product of claim 17, wherein the ellipse parameters are coordinates representing the position of the animal within a plane, the length of the animal's major axis and the length of the animal's minor axis, and the angle at which the animal's head is pointing defined relative to the direction of the major axis.

19. said neural network architecture comprising: predicting a foreground-background segmented image from an input video frame; predicting, in terms of pixels, whether an animal is present in the input video frame based on the segmented image; outputting a segmentation mask based on the pixel-wise prediction; and fitting an ellipse to the portion of the segmentation mask where the animal is predicted to be present to determine values ​​characterizing the predetermined ellipse parameters; 18. The computer program product of claim 17, wherein the computer program product is an encoder-decoder segmentation network configured to:

20. the encoder-decoder segmentation network: a feature encoder configured to abstract the input video frame into a set of small spatial resolution features; a feature decoder configured to transform the set of features into the same shape as the input video frame and output the foreground-background segmented image; an angle predictor configured to predict an angle at which the animal's head is pointing; 20. The computer program product of claim 19, comprising:

21. 18. The method of claim 17, wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of most probable values ​​for each ellipse parameter of the ellipse description.

22. 22. The method of claim 21 , wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, the abstraction being utilized to generate the heatmap.

23. 18. The method of claim 17, wherein the neural network architecture comprises a recurrent network configured to extract features from input video frames and directly predict values ​​characterizing each of the ellipse parameters.

24. 18. The method of claim 17, wherein the animal is a rodent.

25. A system with: It is an arena, Frame, an enclosure attached to the frame and sized to house an animal, the enclosure including a door configured to allow access to an interior of the enclosure; an arena containing; and 1. An acquisition system comprising: camera; at least two sets of light sources, each set of light sources configured to emit light incident on the housing at a different wavelength; at least two sets of light sources configured to capture video data of at least a portion of the housing when the camera is illuminated by at least one of the sets of light sources; in electrical communication with the camera and the plurality of sets of light sources; generating control signals operative to control the acquisition of video data by the camera and the emission of light by the sets of light sources; and receiving video data captured by the camera; a controller configured to: a data storage device in electrical communication with the controller configured to store video data received from the controller; Acquisition system including:

26. 26. The system of claim 25, wherein at least a portion of the housing is substantially opaque to visible light.

27. 26. The system of claim 25, wherein at least a portion of the housing is formed from a material that is substantially opaque to visible light wavelengths.

28. 26. The system of claim 25, wherein at least a portion of the housing is formed from a material that is substantially non-reflective to infrared wavelengths.

29. 26. The system of claim 25, wherein at least a portion of the housing is formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

30. 26. The system of claim 25, wherein the first set of light sources includes one or more first illuminators configured to emit light at one or more visible light wavelengths, and the second set of light sources includes one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

31. 31. The system of claim 30, wherein the wavelength of the infrared light is about 940 nm.

32. 26. The system of claim 25, wherein the camera is configured to capture video data at a resolution of at least 480x480 pixels.

33. 26. The system of claim 25, wherein the camera is configured to capture video data at a frame rate that is higher than the frequency of mouse movements.

34. 26. The system of claim 25, wherein the camera is configured to capture video data at a frame rate of at least 29 frames per second (fps).

35. 26. The system of claim 25, wherein the camera is configured to acquire video data having at least an 8-bit depth.

36. 26. The system of claim 25, wherein the camera is configured to acquire video data at infrared wavelengths.

37. 26. The system of claim 25, wherein the controller is configured to compress video data received from the camera.

38. 38. The system of claim 37, wherein the controller is configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing variance-based background subtraction.

39. 39. The system of claim 38, wherein the MPEG codec filter is Q0 HQDN3D.

40. 31. The system of claim 30, wherein the controller is configured to request the first light source to illuminate the enclosure according to a schedule that simulates a light-dark cycle.

41. 31. The system of claim 30, wherein the controller is configured to request the first light source to illuminate the enclosure with visible light having an intensity of between approximately 50 lux and approximately 800 lux during a light portion of the light-dark cycle.

42. 31. The system of claim 30, wherein the controller is configured to request the second light source to illuminate the enclosure with infrared light such that the temperature rise of the enclosure due to the infrared illumination is less than 5°C.

43. 31. The system of claim 30, wherein the controller is configured to request the first light source to illuminate the enclosure according to 1024 logarithmically scaled levels of illumination.

44. illuminating an enclosure configured to house an animal with at least one set of light sources, each set of light sources configured to emit light at a different wavelength; acquiring, with a camera, video data of at least a portion of the enclosure illuminated by at least one of the sets of light sources; generating, by a controller in electrical communication with the camera and the sets of light sources, control signals operative to control the acquisition of video data by the camera and the emission of light by the sets of light sources; receiving, by the controller, video data captured by the camera; A method comprising:

45. 45. The method of claim 44, wherein at least a portion of the housing is substantially opaque to visible light.

46. 45. The method of claim 44, wherein at least a portion of the housing is formed from a material that is substantially opaque to visible light wavelengths.

47. At least a portion of the housing is formed of a material that is substantially non-reflective to infrared light wavelengths. Item 45. The method according to item 44.

48. 45. The method of claim 44, wherein at least a portion of the housing is formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

49. 45. The method of claim 44, wherein the first set of light sources includes one or more first illuminators configured to emit light at one or more visible light wavelengths, and the second set of light sources includes one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

50. 50. The method of claim 49, wherein the wavelength of the infrared light is about 940 nm.

51. 45. The method of claim 44, wherein the camera is configured to capture video data at a resolution of at least 480x480 pixels.

52. 45. The method of claim 44, wherein the camera is configured to capture video data at a frame rate that is higher than the frequency of mouse movements.

53. 45. The method of claim 44, wherein the camera is configured to capture video data at a frame rate of at least 29 frames per second (fps).

54. 45. The method of claim 44, wherein the camera is configured to acquire video data having at least 8-bit depth.

55. 45. The method of claim 44, wherein the camera is configured to acquire video data at infrared wavelengths.

56. 45. The method of claim 44, wherein the controller is configured to compress video data received from the camera.

57. 57. The method of claim 56, wherein the controller is configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing variance-based background subtraction.

58. 58. The method of claim 57, wherein the MPEG codec filter is Q0 HQDN3D.

59. 50. The method of claim 49, wherein the controller is configured to request the first light source to illuminate the enclosure according to a schedule that simulates a light-dark cycle.

60. 50. The method of claim 49, wherein the controller is configured to request the first light source to illuminate the enclosure with visible light having an intensity of between approximately 50 lux and approximately 800 lux during a light portion of the light-dark cycle.

61. 50. The method of claim 49, wherein the controller is configured to request the second light source to illuminate the enclosure with infrared light such that the temperature of the enclosure increases by less than 5°C due to the infrared illumination.

62. 50. The method of claim 49, wherein the controller is configured to request the first light source to illuminate the enclosure according to 1024 logarithmically scaled levels of illumination. Law.

Images