Long-term continuous animal behavior monitoring

Abstract

This invention provides a system and method for continuously monitoring the behavior of animals such as small rodents. [Solution] The animal tracking method includes the steps of: receiving video data representing the observation of an animal using a processor; and executing a neural network architecture using a processor. The neural network architecture receives input video frames extracted from the video data; generates an elliptic description of at least one animal based on the input video frames; the elliptic description is defined by predetermined elliptic parameters; and provides data for at least one animal that includes values ​​characterizing the predetermined elliptic parameters.

Description

Technical Field

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 542,180, filed Aug. 7, 2017, entitled “Long-Term and Continuous Animal Behavioral Monitoring” and U.S. Provisional Patent Application No. 62 / 661,610, filed Apr. 23, 2018, entitled “Robust Mouse Tracking In Complex Environments Using Neural Networks”. The entire contents of each of these applications are incorporated by reference.

Background Art

[0002]

[0002] Animal behavior can be understood as the output of the nervous system in response to internal or external stimuli. The ability to accurately track an animal can be beneficial as part of the process of classifying animal behavior. For example, changes in behavior are prominent features of aging, mental illness, or metabolic disease, and can reveal important information about the effects on an animal's physiological, neurocognitive, and emotional states.

Summary of the Invention

[0003]

[0003] Conventionally, experiments to evaluate animal behavior have been performed non-invasively, and researchers interact directly with the animals. As an example, a researcher may remove an animal such as a mouse from its living environment (e.g., a cage) and move the animal to a different environment (e.g., a device such as a maze). Then, the researcher may position themselves near the new environment and observe the animal's working ability by tracking the animal. However, it is known that animals may exhibit different behaviors in a new environment or different behaviors towards an experimenter performing a test. This often leads to data confusion, resulting in non-reproducible and misleading results.

[0004]

[0004] Minimally invasive monitoring techniques are being developed to minimize human interference during behavioral monitoring experiments. As an example, video monitoring for use in monitoring animal behavior is being studied. However, challenges remain with video monitoring. On one hand, the main hurdle is being able to continuously capture video data with high spatiotemporal resolution over a long period of time under a set of wide-ranging environmental conditions. Observational studies of animals conducted over long periods such as days, weeks, and / or months can generate large amounts of data that are costly to acquire and store. On the other hand, even assuming that video data of sufficient quality can be acquired and stored, it is not economically feasible for researchers to manually scrutinize the large amount of video footage generated during long-term observations and to track animals over such long periods. This challenge becomes more pronounced when the number of animals being observed increases, as may be required when conducting new drug screening or genomics experiments.

[0005]

[0005] To address this problem, computer-based technologies have been developed to analyze video recordings of animal behavior. However, existing computer-based systems cannot accurately track different animals in complex and dynamic environments. For example, existing computer-based technologies for tracking animals cannot accurately identify a single animal from its background (e.g., cage walls and / or floor, objects in the cage such as a water dish) or to identify multiple animals from each other. At best, if a given animal is not accurately tracked during the observation period, valuable observational data may be lost. At worst, if a given animal or part of it is mistracked or mistaken for something else during the observation period, Furthermore, errors may be introduced into the behavior classified from the acquired video data. Although techniques such as altering animal fur color are employed to facilitate tracking, altering animal fur color may also alter animal behavior. As a result, existing video tracking methods performed in complex and dynamic environments or with genetically heterogeneous animals require a high level of user involvement, thus negating the aforementioned advantages of video observation. For this reason, large-scale and / or long-term animal monitoring experiments remain impractical.

[0006]

[0006] As neuroscience and behavioral science enter the age of massive behavioral data and computational ethology, better techniques for tracking animals are needed to facilitate the classification of animal behavior over long periods in semi-natural and dynamic environments.

[0007]

[0007] Accordingly, systems and methods using neural networks have been developed that can provide robust and scalable tracking of animals (e.g., mice) in open fields. As an example, systems and methods have been provided that facilitate the acquisition of video data of animal movement with high spatiotemporal resolution. This video data can be continuously captured over a long period of time under a set of wide-ranging environmental conditions.

[0008]

[0008] The acquired video data may be used as input to a convolutional neural network architecture for tracking. The neural network may be trained during training to perform tracking under multiple environmental conditions with high robustness and without user intervention adjustment when new environments or animals are presented. Examples of such experimental conditions may include different mouse strains despite different cage environments, as well as different coat colors, body shapes, and behaviors. Thus, embodiments of the present disclosure may facilitate the monitoring of the behavior of a large number of animals over long periods under heterogeneous conditions by facilitating minimally invasive animal tracking.

[0009]

[0009] In certain embodiments, the disclosed video observation and animal tracking techniques may be employed in combination. However, it should be understood that each of these techniques may be employed individually or in any combination with each other or other techniques.

[0010]

[0010] In one embodiment, a method for tracking animals is provided. This method may include the steps of: having a processor receive video data representing an observation of an animal; and having the processor execute a neural network architecture. The neural network architecture may be configured to receive input video frames extracted from the video data; generate an elliptic description of at least one animal based on the input video frames, the elliptic description being defined by predetermined elliptic parameters; and provide data for at least one animal, including values ​​characterizing the predetermined elliptic parameters.

[0011]

[0011] In another embodiment of this method, the elliptic parameters may be coordinates representing the position of the animal in the plane, the length of the animal's major axis and minor axis, and the angle to which the animal's head is facing, defined with respect to the direction of the major axis.

[0012]

[0012] In another embodiment of this method, an encoder-decoder segmentation network may be used as the neural network architecture. The encoder-decoder segmentation network may be configured to predict a foreground-background segmented image from an input video frame, predict whether an animal is present in the input video frame based on the segmented image in terms of pixels, output a segmentation mask based on the prediction in terms of pixels, and fit an ellipse to the portion of the segmentation mask in which the animal is predicted to be present to determine values ​​that characterize predetermined ellipse parameters.

[0013]

[0013] In another embodiment of this method, the encoder-decoder segmentation network may comprise a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract the input video frame into a set of small spatial resolution features. The feature decoder may be configured to transform the set of features into the same shape as the input video frame and output a foreground-background segmented image. The angle predictor may be configured to predict the angle at which the animal's head is facing.

[0014]

[0014] In another embodiment of this method, the neural network architecture may comprise a binning classification network configured to predict a heatmap of the most probable values ​​for each elliptic parameter of the elliptic description.

[0015]

[0015] In another embodiment of this method, the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, and the abstraction may be employed to generate a heatmap.

[0016]

[0016] In another embodiment of this method, the neural network architecture may comprise a regression network configured to extract features from input video frames and directly predict the values ​​that characterize each elliptic parameter.

[0017]

[0017] In another embodiment of this method, the animal can be a rodent.

[0018] In one embodiment, an animal tracking system is provided. This system may include a data storage device that maintains video data representing observations of animals. The system may also include a processor configured to receive video data from the data storage device and to implement a neural network architecture. The neural network architecture may be configured to receive input video frames extracted from the video data, generate an elliptic description of at least one animal based on the video frames, the elliptic description being defined by predetermined elliptic parameters, and provide data for at least one animal containing values ​​characterizing the predetermined elliptic parameters.

[0018]

[0019] In another embodiment of this system, the elliptic parameters may be coordinates representing the position of the animal in the plane, the length of the animal's major axis and minor axis, and the angle to which the animal's head is facing, defined with respect to the direction of the major axis.

[0019]

[0020] In another embodiment of this system, an encoder-decoder segmentation network can be the neural network architecture. The encoder-decoder segmentation network may be configured to predict a foreground-background segmented image from an input video frame, predict whether an animal is present in the input video frame based on the segmented image in terms of pixels, output a segmentation mask based on the prediction in terms of pixels, and fit an ellipse to the portion of the segmentation mask in which the animal is predicted to be present to determine values ​​that characterize predetermined ellipse parameters.

[0020]

[0021] In another embodiment of this system, the encoder-decoder segmentation network may comprise a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract the input video frame into a set of small spatial resolution features. The feature decoder may be configured to transform the set of features into the same shape as the input video frame and output a foreground-background segmented image. The angle predictor may be configured to predict the angle at which the animal's head is facing.

[0021]

[0022] In another embodiment of this system, the neural network architecture may include a binning classification network. The binning classification network may be configured to predict a heatmap of the most probable values ​​for each elliptic parameter in the elliptic description.

[0022]

[0023] In another embodiment of this system, the binning classification network comprises a feature encoder configured to abstract input video frames to a smaller spatial resolution, and the abstraction may be employed to generate a heatmap.

[0023]

[0024] In another embodiment of this system, the neural network architecture may comprise a regression network configured to extract features from input video frames and directly predict the values ​​characterizing each elliptic parameter.

[0024]

[0025] In another embodiment of this system, the animal can be a rodent.

[0026] In one embodiment, a non - transient computer program product storing instructions is provided. The instructions can execute a method that includes, when executed by at least one data processor of at least one computing system, receiving video data representing an observation of an animal, and executing a neural network architecture. The neural network architecture can be configured to receive an input video frame extracted from the video data, generate an ellipse description of at least one animal based on the input video frame, where the ellipse description is defined by predetermined ellipse parameters, and provide data including values characterizing the predetermined ellipse parameters for at least one animal.

[0025]

[0027] In another embodiment, the ellipse parameters can be coordinates representing the position of the animal in a plane, the lengths of the major and minor axes of the animal, and the angle at which the animal's head is facing, the angle being defined with respect to the direction of the major axis.

[0026]

[0028] In another embodiment, the neural network architecture can be an encoder - decoder segmentation network. The encoder - decoder segmentation network can be configured to predict a foreground - background segmentation image from the input video frame, predict whether an animal is present in the input video frame from the perspective of pixels based on the segmentation image, output a segmentation mask based on the prediction from the perspective of pixels, and fit an ellipse to the portion of the segmentation mask where an animal is predicted to be present to determine values characterizing the predetermined ellipse parameters.

[0027]

[0029] In another embodiment, the encoder-decoder segmentation network may include a feature encoder, a feature decoder, and an angle predictor. The feature encoder may be configured to abstract an input video frame into a set of features with a small spatial resolution. The feature decoder may be configured to convert a set of features into the same shape as the input video frame and output a foreground-background segmentation image. The angle predictor may be configured to predict the angle at which the animal's head is facing.

[0028]

[0030] In another embodiment, the neural network architecture may include a binning classification network configured to predict a heatmap of the most accurate value of each ellipse parameter of the ellipse description.

[0029]

[0031] In another embodiment, the binning classification network may include a feature encoder configured to abstract an input video frame into a small spatial resolution, and the abstraction may be employed to generate a heatmap.

[0030]

[0032] In another embodiment, the neural network architecture may include a regression network configured to extract features from an input video frame and directly predict a value characterizing each ellipse parameter.

[0031]

[0033] In another embodiment, the animal may be a rodent.

[0034] ​In one embodiment, a system is provided which may comprise an arena and an acquisition system. The arena may include a frame and a housing attached to the frame. The housing may include a door sized to accommodate an animal and configured to allow access to the interior. The acquisition system may include a camera, at least two sets of light sources, a controller, and a data storage device. Each set of light sources may be configured to emit light incident on the housing at different wavelengths from each other. The camera may be configured to acquire video data of at least a portion of the housing when illuminated by at least one of the multiple sets of light sources. The controller may be in electrical communication with the camera and the multiple sets of light sources. The controller may be configured to generate control signals that operate to control the acquisition of video data by the camera and the emission of light by the multiple sets of light sources, and to receive the video data acquired by the camera. The data storage device may be in electrical communication with the controller and configured to store the video data received from the controller.

[0032]

[0035] In another embodiment of this system, at least a portion of the housing may be substantially opaque to visible light.

[0036] In another embodiment of this system, at least a portion of the housing may be formed of a material that is substantially opaque to visible light wavelengths.

[0033]

[0037] In another embodiment of this system, at least a portion of the housing may be formed of a material that is substantially non-reflective to infrared light wavelengths.

[0038] In another embodiment of this system, at least a portion of the housing may be formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

[0034]

[0039] In another embodiment of this system, a first set of light sources may include one or more first illuminations configured to emit light at one or more visible light wavelengths, and a second set of light sources may include one or more second illuminations configured to emit light at one or more infrared (IR) light wavelengths.

[0035]

[0040] In another embodiment of this system, the wavelength of infrared light may be approximately 940 nm.

[0041] In another embodiment of this system, the camera may be configured to acquire video data at a resolution of at least 480 x 480 pixels.

[0036]

[0042] In another embodiment of this system, the camera may be configured to acquire video data at a frame rate higher than the frequency of mouse movements.

[0043] In another embodiment of this system, the camera may be configured to acquire video data at a frame rate of at least 29 frames per second (fps).

[0037]

[0044] In another embodiment of this system, the camera may be configured to acquire video data having at least 8 bits of depth.

[0045] In another embodiment of this system, the camera may be configured to acquire video data at infrared wavelengths.

[0038]

[0046] In another embodiment of this system, the controller may be configured to compress the video data received from the camera.

[0047] In another embodiment of this system, the controller may be configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing distributed-based background subtraction.

[0039]

[0048] In another embodiment of this system, Q0 HQDN3D can be used as the filter for the MPEG codec.

[0049] In another embodiment of this system, the controller may be configured to request the first light source to illuminate the housing according to a schedule that simulates a light-dark cycle.

[0040]

[0050] In another embodiment of this system, the controller may be configured to request the first light source to illuminate the housing with visible light having an intensity of approximately 50 lux to approximately 800 lux during the light portion of the light-dark cycle.

[0041]

[0051] In another embodiment of this system, the controller may be configured to request a second light source to irradiate the housing with infrared light such that the temperature rise of the housing due to infrared irradiation is less than 5°C.

[0042]

[0052] In another embodiment of this system, the controller may be configured to instruct the first light source to illuminate the housing according to 1024 levels of illumination scaled logarithmically.

[0043]

[0053] In one embodiment, a method is provided which may include the step of illuminating an enclosure configured to house an animal with at least one set of light sources. Each set of light sources may be configured to emit light of different wavelengths from each other. The method may also include the step of acquiring video data of at least a portion of the enclosure illuminated by at least one of the multiple sets of light sources using a camera. The method may also include the step of generating control signals that operate to control the acquisition of video data by the camera and the emission of light by the multiple sets of light sources using a controller electrically connected to the camera and the multiple sets of light sources. Furthermore, the method may include the step of receiving the video data acquired by the camera using the controller.

[0044]

[0054] In another embodiment of this method, at least a portion of the housing may be substantially opaque to visible light.

[0055] In another embodiment of this method, at least a portion of the housing may be formed of a material that is substantially opaque to visible light wavelengths.

[0045]

[0056] In another embodiment of this method, at least a portion of the housing may be formed of a material that is substantially non-reflective to infrared light wavelengths.

[0057] In another embodiment of this method, at least a portion of the housing may be formed from a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

[0046]

[0058] In another embodiment of this method, a first set of light sources may include one or more first illuminations configured to emit light at one or more visible light wavelengths, and a second set of light sources may include one or more second illuminations configured to emit light at one or more infrared (IR) light wavelengths.

[0047]

[0059] In another embodiment of this method, the wavelength of infrared light may be approximately 940 nm.

[0060] In another embodiment of this method, the camera has at least 480 x 480 pixels It can be configured to acquire video data at a specific resolution.

[0048]

[0061] In another embodiment of this method, the camera may be configured to acquire video data at a frame rate higher than the frequency of mouse movements.

[0062] In another embodiment of this method, the camera may be configured to acquire video data at a frame rate of at least 29 frames per second (fps).

[0049]

[0063] In another embodiment of this method, the camera may be configured to acquire video data having at least 8 bits of depth.

[0064] In another embodiment of this method, the camera may be configured to acquire video data at infrared wavelengths.

[0050]

[0065] In another embodiment of this method, the controller may be configured to compress the video data received from the camera.

[0066] In another embodiment of this method, the controller may be configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing distributed-based background subtraction.

[0051]

[0067] In another embodiment of this method, Q0 HQDN3D can be used as the filter for the MPEG codec.

[0068] In another embodiment of this method, the controller may be configured to instruct the first light source to illuminate the housing according to a schedule that simulates a light-dark cycle.

[0052]

[0069] In another embodiment of this method, the controller may be configured to require the first light source to illuminate the housing with visible light having an intensity of approximately 50 lux to approximately 800 lux during the light portion of the light-dark cycle.

[0053]

[0070] In another embodiment of this method, the controller may be configured to instruct a second light source to irradiate the housing with infrared light such that the temperature rise of the housing due to infrared irradiation is less than 5°C.

[0054]

[0071] In another embodiment of this method, the controller may be configured to instruct the first light source to illuminate the housing according to 1024 levels of illumination scaled logarithmically.

[0055]

[0072] The above and other features will be easily understood from the following detailed explanation, which is accompanied by the attached drawings. [Brief explanation of the drawing]

[0056] [Figure 1]

[0073] Figure 1 is a flowchart illustrating one exemplary embodiment of the operating environment for animal tracking. [Figure 2]

[0074] Figure 2 is a schematic diagram of one embodiment of an animal behavior monitoring system. [Figure 3]

[0075] Figures 3A-3F are images showing sample frames acquired by the system in Figure 2; (A-C) visible light; (D-F) infrared (IR) light. [Figure 4]

[0076] Figures 4A-4B are plots of quantum efficiency as a function of wavelength for two camera models; (A) relative response for Sentech STC-MC33USB; (B) quantum efficiency for Basler acA1300-60gm-NIR. [Figure 5]

[0077] This figure shows a plot of the transparency-wavelength profile of an IR long-pass filter. [Figure 6]

[0078] Figures 6A–6D are images illustrating exemplary embodiments of video frames to which different compression techniques have been applied; (A) Uncompressed; (B) MPEG4 Q0; (C) MPEG4 Q5; (D) MPEG4 Q0 HQDN3D; [Figure 7]

[0079] Figure 7 shows an embodiment of the components of an acquisition system suitable for use with the system in Figure 2. [Figure 8A]

[0080] Figure 8A is a schematic diagram of an exemplary embodiment of an observation environment analyzed in accordance with this disclosure, including black mice, gray mice, albino mice, and mottled mice. [Figure 8B]

[0081] Figure 8B is a schematic diagram illustrating a situation where animal tracking is insufficient. [Figure 8C]

[0082] Figure 8C is a schematic diagram of an exemplary embodiment of mouse tracking, including tracking an elliptical object. [Figure 9]

[0083] Figure 9 is a schematic diagram of an exemplary embodiment of a segmentation network architecture. [Figure 10]

[0084] Figure 10 is a schematic diagram of an exemplary embodiment of a binning classification network architecture. [Figure 11]

[0085] Figure 11 is a schematic diagram of an exemplary embodiment of a regression classification network architecture. [Figure 12A]

[0086] Figure 12A shows an exemplary embodiment of a graphical user interface that illustrates the arrangement of two marks, a foreground (F) and a background (B). [Figure 12B]

[0087] Figure 12B is a diagram illustrating an exemplary embodiment of a graphical user interface that shows segmentation as a result of the marking in Figure 12A. [Figure 13A]

[0088] Figure 13A shows plots of the training curves for the segmentation, regression, and binning classification network embodiments shown in Figures 9 to 11. [Figure 13B]

[0089] Figure 13B shows plots of validation curves for the segmentation, regression, and binning classification network embodiments shown in Figures 9-11. [Figure 13C]

[0090] Figure 13C shows a plot of the training and validation performance of the segmentation network architecture shown in Figure 9. [Figure 13D]

[0091] Figure 13D shows plots of the training and validation performance of the regression network architecture in Figure 11. [Figure 13E]

[0092] Figure 13E shows plots of the training and validation performance of the binning classification network architecture shown in Figure 10. [Figure 14A]

[0093] Figure 14A is a plot of the training error as a function of the number of training steps for multiple sets of different sizes, according to an embodiment of the present disclosure. [Figure 14B]

[0094] Figure 14B is a plot of the validation error as a function of the training steps for multiple sets of different sizes, according to an embodiment of the present disclosure. [Figure 14C]

[0095] Figure 14C shows a plot of training and validation errors as a function of steps for the full training set of training samples. [Figure 14D]

[0096] Figure 14D shows a plot of training and validation errors as a function of steps for a training set containing 10,000 (10k) training samples. [Figure 14E]

[0097] Figure 14E shows a plot of training and validation errors as a function of steps for a training set containing 5,000 (5k) training samples. [Figure 14F]

[0098] Figure 14F shows a plot of training and validation errors as a function of steps for a training set containing 2,500 (2.5k) training samples. [Figure 14G]

[0099] Figure 14G shows a plot of training and validation errors as a function of steps for a training set containing 1,000 (1k) training samples. [Figure 14H]

[0100] Figure 14H shows a plot of training and validation errors as a function of steps for a training set containing 500 training samples. [Figure 15]

[0101] Figures 15A-15D are frames of captured video data with color indicators superimposed to distinguish each mouse from the others; (A-B) visible light illumination; (C-D) infrared light illumination. [Figure 16]

[0102] Figure 16 is a plot comparing the performance of the segmentation network architecture shown in Figure 9 with that of a beam break system. [Figure 17]

[0103] Figure 17A shows one embodiment of the present disclosure and a plot of predictions by Ctrax.

[0057]

[0104] Figure 17B shows the segmentation network architecture from Figure 9. This is a plot of the relative standard deviation of the determined short-axis predictions. [Figure 18A]

[0105] Figure 18A is a plot of the total distance tracked for a large-scale strain study of genetically distinct animals, determined by the segmentation network architecture in Figure 9. [Figure 18B]

[0106] Figure 18B is a plot of circadian movement patterns observed in six animals continuously tracked over four days in a dynamic environment determined by the segmentation network architecture in Figure 9. [Modes for carrying out the invention]

[0058]

[0107] Note that the drawings are not necessarily proportional to the actual size. The drawings are disclosed herein. Since this disclosure is intended to show only representative aspects of the subject matter, it should not be considered to limit the scope of this disclosure.

[0059]

[0108] For clarity, in this specification, with respect to small rodents such as mice, one animal is used. Alternatively, exemplary embodiments of systems and corresponding methods for facilitating behavioral monitoring by video capturing multiple animals and tracking one or more animals are discussed. However, embodiments of the disclosure may be employed and / or configured to monitor other animals without limitation.

[0060]

[0109] Figure 1 shows Arena 200, Acquisition System 700, and Neural Network This is a schematic diagram illustrating an exemplary embodiment of an operating environment 100 comprising a tracking system configured to implement a tracker. One or more mice may be housed in the arena 200, as will be discussed in more detail below. Video data of at least one animal (such as a mouse) is acquired. The video data may be acquired alone or in combination with other data relating to animal monitoring, such as audio and environmental parameters (e.g., temperature, humidity, light intensity). The process of acquiring this data, including the control of cameras, microphones, lighting, other environmental sensors, data storage, and data compression, may be performed by the acquisition system 700. The acquired video data may be input to a tracking system capable of running a convolutional neural network (CNN) to track one or more animals based on this video data.

[0061] I. Video Data Acquisition

[0110] In one embodiment, a system captures video data including the movement of an animal. A method is provided. As discussed below, video data is continuously acquired over a predetermined period (for example, one minute or more, one hour or more, one day or more, one week or more, one month or more, one year or more, etc.). This is possible. The characteristics of the video data can be sufficient to facilitate subsequent analysis for the extraction of behavioral patterns, and include, but are not limited to, one or more of the following: resolution, frame rate, and bit depth. Practical solutions are provided that appear to be more robust and of higher quality than existing video capture systems. Embodiments of this disclosure are tested in several ways for visually marking mice. Practical examples of synchronized acquisition of video and ultrasonic vocalization data are also presented.

[0062]

[0111] In one embodiment, animal monitoring is performed over a period of approximately 4 to 6 weeks. A video monitoring system may be deployed for this purpose. Deployment may include one or more of the following: image acquisition and arena design, fine-tuning of chamber design, development of video acquisition software, acquisition of audio data, load testing of cameras, chambers, and software, and determination of chamber production during the deployment phase. Each of these is described in detail below. The aforementioned 4-6 week observation period is given for illustrative purposes, and embodiments of the present disclosure may be adopted for longer or shorter periods as needed.

[0063] a. Arena design

[0112] Proper arena design can be crucial for obtaining high-quality behavioral data. An arena is an animal's "dwelling" and can be configured to provide one or more of the following: isolation from environmental disturbances, adequate circadian lighting, food, water, and bedding. It is also generally a stress-free environment.

[0064]

[0113] From a behavioral standpoint, the arena minimizes stress and environmental disturbances. Ideally, it should be able to express natural behavior.

[0114] From a breeding perspective, the arena requires cleaning, adding or removing items, removing mice, and feeding. It would also be desirable to facilitate the addition and removal of water.

[0065]

[0115] From a veterinary perspective, Arena does not substantially hinder interest behavior, and health In addition to providing diagnosis and treatment, it would be desirable to facilitate the monitoring of environmental conditions (e.g., temperature, humidity, light, etc.).

[0066]

[0116] From a computer vision perspective, the arena exhibits substantial occlusion and distortion. It is desirable to facilitate the acquisition of high-quality video and audio without reflection, and / or noise pollution, and without substantially hindering the expression of interest behavior.

[0067]

[0117] From a facility standpoint, the arena effectively minimizes floor space and also... It would be desirable to provide relatively easy storage that does not require disassembly or reassembly.

[0068]

[0118] Therefore, the arena brings about a balance between behavior, breeding, calculation, and facilities. It can be configured in such a way. An exemplary embodiment of the arena 200 is shown in Figure 2. The arena 200 may comprise a frame 202 on which a housing 204 is mounted. The housing 204 may comprise a door 206 configured to allow access to the interior. One or more cameras 210 and / or lighting 212 may be mounted adjacent to the frame 202 (for example, above the housing 204) or directly to the frame 202.

[0069]

[0119] As will be discussed in detail below, in certain embodiments, the lighting 212 is less It may include at least two sets of light sources. Each set of light sources may include one or more illuminations configured to emit light incident on the housing 204 at a different wavelength than the other set. For example, A first set of light sources may be configured to emit light at one or more visible wavelengths (for example, approximately 390 nm to approximately 700 nm), and a second set of light sources may be configured to emit light at one or more infrared (IR) wavelengths (for example, above approximately 700 nm to approximately 1 mm).

[0070]

[0120] Camera 210 and / or lighting 212 are connected to the user interface 214 and They can be electrically connected. The user interface 214 can be a display configured to display video data acquired by the camera 210. In certain embodiments, the user interface 214 can be a touchscreen display configured to display one or more user interfaces for controlling the camera 210 and / or the lighting 212.

[0071]

[0121] As an alternative or addition to the above, camera 110, lighting 212, and user input The surface 214 may be electrically connected to the controller 216. The controller 216 may be configured to generate control signals that operate to control the acquisition of video data by the camera 210, the emission of light by the illumination 212, and / or the display of the acquired video data by the user interface 214. In certain embodiments, the user interface may be optionally omitted.

[0072]

[0122] Furthermore, the controller 216 can communicate with the data storage device 220. The controller 216 may be configured to receive video data acquired by the camera 210 and transmit and store the acquired video data in the data storage device 220. Communication between one or more of the camera 210, lighting 212, user interface 214, controller 216, and data storage device 220 may be performed using a wired communication link, a wireless communication link, or a combination thereof.

[0073]

[0123] As discussed below, Arena 200 is designed for behavior, breeding, calculation, and facilities. It may have an open-field design configured to achieve the desired balance while enabling completion within a predetermined period (for example, approximately 5 months).

[0074] material

[0124] In certain embodiments, the housing 204 (for example, the lower part of the housing 204) is constructed At least a portion of the materials constituting the enclosure may be substantially opaque to visible light wavelengths. In this way, in addition to visible light emitted by light sources other than illumination 212, visual stimuli that an animal inside the enclosure 204 can observe (e.g., movement of objects and / or the user) may be suppressed and / or substantially eliminated. In an additional embodiment, the materials constituting the enclosure 204 may be substantially non-reflective to infrared wavelengths to facilitate the acquisition of video data. The thickness of the walls of the enclosure 204 may be selected within a range suitable for providing mechanical support (e.g., approximately 0.3175 cm (1 / 8 inch) to approximately 0.635 cm (1 / 4 inch)).

[0075]

[0125] In one embodiment, the housing 204 is made of polyvinyl chloride (PVC) or polio It can be constructed using foamed sheets formed from ximethylene (POM). An example of POM is Delrin® (DuPont, Wilmington, DE, USA). Such foamed sheets are beneficial because they can provide sufficient versatility and durability for long-term animal monitoring of Arena 200.

[0076]

[0126] In one embodiment, the frame 202 has a plurality of legs 202a and the space between them It may include one or more shelf sections 202b extending (for example, horizontally). As an example, the frame 202 can be a commercially available shelving system of a predetermined size with fixed wheels for moving into the storage area. In one embodiment, the predetermined size is approximately 61c A size of m(2 feet) x 61 cm(2 feet) x 183 cm(6 feet) is possible (for example, Super Erecta Metroseal 3®, InterMetro Industries Corporation, Wilkes-Barre, PA, USA). However, in other embodiments, arenas of different sizes may be employed without limitation.

[0077] b. Data acquisition

[0127] The video acquisition system includes a camera 210, lighting 212, and a user interface. The system may comprise a 214, a controller 216, and a data storage device 220. The video acquisition system may be designed to have a predetermined balance of performance characteristics. These performance characteristics include, but are not limited to, the video acquisition frame rate, bit depth, resolution of each frame, and spectral sensitivity in the infrared region, as well as one or more of video compression and storage. As will be discussed below, these parameters may be optimized to maximize data quality while minimizing data volume.

[0078]

[0128] In one embodiment, the camera 210 has a resolution of approximately 640 x 480 pixels. Video data having at least one of approximately 29 fps and approximately 8-bit depth can be acquired. Using these video acquisition parameters, approximately 33 GB / hour of uncompressed video data can be generated. As an example, the camera 210 can be a Sentech USB2 (Sensor Technologies America, Inc., Carrollton, TX, USA). Figures 3A to 3F show sample frames acquired from one embodiment of the video acquisition system using visible light (Figures 3A to 3C) and infrared (IR) light (Figures 3D to 3F).

[0079]

[0129] As discussed below, the collected video data is from camera 210 and / or Alternatively, it can be compressed by the controller 216.

[0130] In another embodiment, the video acquisition system determines the resolution of the acquired video data It can be configured to approximately double the resolution (for example, to about 960 x 960 pixels). Four other cameras with higher resolution than the Sentech USB were investigated, as shown below.

[0080] [Table 1]

[0081]

[0131] These cameras are cost, resolution, maximum frame rate, bit depth, and They can differ in terms of quantum efficiency.

[0132] The video acquisition system is a monochrome, approximately 30fps, and It can be configured to collect 8-bit depth video data. According to the Shannon-Nyquist theorem, the frame rate should be at least twice the frequency of the events of interest (see, e.g., Shannon (1994)). Mouse behavior can vary from a few hertz for grooming to 20 hertz for rapid movement (e.g., Deschenes See et al. (2012), Kalueff et al. (2010), and Wiltschko et al. (2015). Since grooming has been observed to occur at a maximum of approximately 7 Hz, it is considered appropriate to record video at a frame rate higher than the frequency of mouse movement (e.g., approximately 29 fps) to observe most mouse behaviors. However, cameras can rapidly lose sensitivity in the IR region. This loss of contrast can be overcome by increasing the level of IR light, but increasing the intensity of IR light may raise the ambient temperature.

[0082] illumination

[0133] As described above, the illumination 212 emits one or more types of light, such as visible white light and infrared light. It can be configured to emit light of a certain type. Visible light is employed for illumination and can be programmed (e.g., by controller 216) to provide light-dark cycles and adjustable intensity. The ability to adjust the illumination cycle allows for the simulation of sunlight exposure that animals receive in the wild. The length of the light-dark period can be adjusted to simulate seasons, and illumination shifts can be performed to simulate jet lag (circadian phase advance and retreat) experiments. High-intensity illumination can also be employed to induce anxiety in certain animals, and low-intensity illumination can be employed to elicit different exploratory behaviors. Thus, the ability to temporally control the length of light-dark and light intensity is essential for proper behavioral experiments.

[0083]

[0134] In certain embodiments, the controller 216 controls the light portion of the light-dark cycle. The housing 204 is then illuminated with visible light having an intensity of approximately 50 lux to approximately 800 lux. The system may be configured to require a visible light source. The selected light intensity may vary depending on the type of movement of the object being observed. In one embodiment, a relatively low intensity (e.g., approximately 200 lux to approximately 300 lux) may be used to encourage and observe exploratory movements by mice.

[0084]

[0135] In certain embodiments, by using an IR long-pass filter, In the red region, almost all video data can be acquired by camera 210. The IR long-pass filter can remove almost all visible light input to camera 210. IR light is beneficial because it enables uniform illumination of the housing 104 day and night.

[0085]

[0136] Two wavelengths of IR light (850nm and 940nm LEDs) were evaluated. 850nm light exhibits a vivid red hue visible to the naked eye and can result in low-luminosity exposure for animals. However, such dim light can induce emotional fluctuations in mice. Therefore, 940nm light is chosen for recording.

[0086]

[0137] Recording at a wavelength of 940 nm can result in very low quantum yields in the camera. Due to the high gain, this can result in a coarse-looking image. Therefore, various infrared illumination levels using different cameras were evaluated to identify the maximum light level obtainable without substantially raising the temperature of the housing 204 for infrared illumination. In certain embodiments, the temperature of the housing 204 can be raised by only about 5°C or less (for example, about 3°C ​​or less).

[0087]

[0138] The Basler acA1300-60gm-NIR camera was also evaluated. This camera has approximately 3 to 4 times the spectral sensitivity at 940 nm compared to the other cameras listed in Table 1, as shown in Figures 4A and 4B. Figure 4A shows the spectral sensitivity of the Sentech camera as a representative example in terms of relative response, and Figure 4B shows the spectral sensitivity of the Basler camera in terms of quantum efficiency. Quantum efficiency is a measure of electrons emitted in relation to photons colliding with the sensor. Relative response is the quantum efficiency expressed on a scale of 0 to 1. In Figures 4A and 4B, the wavelength of 940 nm is further shown as a perpendicular line for reference.

[0088]

[0139] The visible light cycle provided by illumination 212 is controlled by controller 216 or It may be controlled by another device communicating with the illumination 212. In certain embodiments, the controller 216 may comprise an illumination control panel (Phenome Technologies, Skokie, IL). The control panel is logarithmically scaled, controllable via an RS485 interface, and has 1024 levels of illumination capable of performing dawn / dusk events. As will be discussed in more detail below, the control of visible light may be incorporated into control software executed by the controller 216.

[0089] filter

[0140] As mentioned above, as an optional choice, during video data acquisition, almost all visible light is captured by the camera. To prevent the light level from reaching 210, an IR long-pass filter may be employed. For example, a physical IR long-pass filter may be used in conjunction with camera 110. This configuration can provide substantially uniform illumination regardless of the light and dark phase of arena 200.

[0090]

[0141] Filters potentially suitable for use in embodiments of the disclosed systems and methods Profiles are shown in Figure 5 (for example, IR pass filters 092 and 093). An IR cut filter 486 that blocks IR light is shown for comparison. Additional profiles for RG-850 (glass, Edmunds Optics) and 43-949 (plastic, laser curable, Edmunds Optics) are also considered suitable.

[0091] lens

[0142] In one embodiment, the camera lens is 0.847 cm (1 / 3"), A 3.5-8mm, f1.4 (CS mount) lens is possible. This lens can produce the images shown in Figures 3A and 3B. Similar C-mount lenses can also be used.

[0092] Video compression

[0143] Ignoring compression, camera 210 produces approximately 1MB / frame, approximately Raw video data can be generated at a rate of approximately 30 MB / second, 108 GB / hour, or 2.6 TB / day. When choosing a storage method, various objectives are possible. Depending on the video, removing specific elements of the video before long-term storage may be a beneficial option. Furthermore, when considering long-term storage, applying filters or other forms of processing (e.g., by controller 216) should be desirable. However, if the processing method is to be changed later, preserving the original video data, i.e., raw video data, may be a beneficial solution. An example of a video compression test is described below.

[0093]

[0144] Pixel resolution approximately 480x480, approximately 29fps, and approximately 8-bit. Multiple compression standards were evaluated for video data collected at / pixel for approximately 100 minutes. The two lossless formats tested from the raw video were Dirac and H264. H264 has a slightly smaller file size but requires a slightly longer time for code conversion. Dirac may be more widely favored due to subsequent code conversion to other formats.

[0094]

[0145] The MPEG4 lossy format was also evaluated because it is closely related to H264. It is known that the bitrate can be well controlled. There are two methods for setting the bitrate. The first is to set a constant bitrate throughout the entire encoded video, and the second is to set a variable bitrate based on deviations from the original video. In ffmpeg using the MPEG4 encoder, setting a variable bitrate can be easily achieved by selecting a quality value (0 to 31 (0 is virtually lossless)).

[0095]

[0146] In Figures 6A to 6D, three frames are taken for the original (raw) captured video frame. Different image compression methods are compared. The original image is shown in Figure 6 The other three methods are shown in Figure 6 B~Figure 6 In D, the difference in pixels from the original image is shown, indicating only the effect of compression. That is, the compressed image differs significantly from the original image. Therefore, smaller differences are better, and higher compression ratios are better. (Figure) 6 As shown in Figure B, the compression performed according to the MPEG4 codec with the Q0 filter exhibits a compression ratio of 1 / 17. 6 As shown in Figure C, the compression performed according to the MPEG4 codec with the Q5 filter exhibits a compression ratio of 1 / 237. 6 As shown in D, the compression performed according to the MPEG4 codec with the HQDN3D filter exhibits a compression ratio of 1 / 97.

[0096]

[0147] Video data collected according to the disclosed embodiments includes quality parameter (Q0 Filter (Figure) 6 B) Q0 HQDN3D filter (Figure) 6 When using D), approximately 0.01% of pixels are altered from the original image (intensity increases or decreases by up to 4%). This accounts for approximately 25 pixels per frame. The majority of these pixels are located at the boundaries of shading. Naturally, this small image alteration follows the scale of noise interfering with the camera 210 itself. Higher quality values ​​(e.g., Q5 (Figure)) 6 In C)), artifacts may be introduced to better compress the video data. These often lead to artifacts involving block noise that appear when care is not taken during compression.

[0097]

[0148] In addition to these formats, to accommodate individual user datasets, Other suitable lossless formats can be generated. Two of these are the FMF codec (fly movie format) and the UFMF codec (micro fly movie format). The purpose of these formats is to minimize irrelevant information and optimize readability for tracking. Because these formats are lossless and operate on a fixed background model, no substantial data compression was possible due to unfiltered sensor noise. The results of this compression evaluation are shown in Table 2.

[0098] [Table 2]

[0099]

[0149] In addition to selecting a codec for data compression, reducing background noise in images is also desirable. This is likely the case. Background noise is inherent in all cameras and is often called dark noise, representing the baseline noise in the image.

[0100]

[0150] To eliminate this noise, extend the exposure time, widen the aperture, and reduce the gain. There are many methods, however, if these methods directly affect the experiment, they are not viable options. Therefore, the ffmpeg HQDN3D filter, which incorporates spatiotemporal information and removes small fluctuations, may be employed.

[0101]

[0151] As shown in Figures 6B to 6D, the HQDN3D filter acquires video data A significant reduction in file size is observed (for example, approximately 100 times smaller compared to the original video data file size). MPE with HQDN3D filter After compression with the G4 codec, the resulting average bitrate for compressed video is approximately 0.34 GB / hour. Furthermore, it has been experimentally verified that virtually all information loss is less than a few orders of magnitude greater than that generated from sensor noise (video acquired without a mouse). This type of noise reduction significantly improves compressibility.

[0102]

[0152] Unexpectedly, the HQDN3D filter is a convolutional nucleus, which will be discussed in detail below. It has been found that this significantly improves the tracking performance of neural networks (CNNs). Without theoretical constraints, this performance improvement is thought to be achieved because the HQDN3D filter is a variance-based background subtraction method. Low variance makes foreground identification easier, resulting in high-quality tracking.

[0103] Ultrasonic Audio Acquisition

[0153] Mice use vocalizations in the ultrasonic range for social communication, mating, It can be used for aggression and nurturing (see, e.g., Grimsley et al. (2011)). Along with olfactory and tactile stimuli, this vocalization can be one of the most prominent forms of communication in mice. Although not tested in mice, in humans, changes in voice and vocalization (aging) may indicate transitions such as puberty and aging (see, e.g., Decoster and Debruyne (1997), Martins et al. (2014), Mueller (1997)).

[0104]

[0154] Therefore, as will be discussed in detail below, the embodiment of Arena 200 is 1 The system may further comprise one or more microphones 222. The microphones 222 may be mounted on the frame 202 and configured to acquire audio data from animals placed in the housing 204. Synchronized data acquisition can be achieved by using the microphones 222 in the form of a microphone array. This configuration of microphones 222 allows for the identification of vocalizing mice. The ability to further determine vocalizing mice within a group of mice has been demonstrated in recent years using microphone arrays (see, for example, Heckman et al. (2017) and Neunuebel et al. (2015)).

[0105]

[0155] Similar to Neunuebel et al., a data collection setup is provided. It is possible. Four microphones can be positioned on the side of the arena where sound can be captured. When integrated with video data, the vocalizing mouse can be identified using maximum likelihood (see, for example, Zhang et al. (2008)).

[0106] Environmental sensors

[0156] In one embodiment, Arena 200 controls temperature, humidity, and / or light intensity. The system may further comprise one or more environmental sensors 224 configured to measure one or more environmental parameters, such as visible and / or IR. In certain embodiments, the environmental sensors 224 may be integrated and configured to measure two or more environmental parameters (see, for example, Phenome Technologies, Skokie, IL). The environmental sensors 224 can be electrically connected to the controller 216 to collect daily temperature and humidity data along with light levels. The collected environmental data, including minimum and maximum temperatures, may be output for display in a user interface indicating lighting conditions (see the description of the control software below).

[0107] Software-controlled system

[0157] Software control by controller 216 for data acquisition and optical control. The system can be run. The software control system can be configured to independently collect video, audio / ultrasonic, and environmental data, along with corresponding timestamps. Thus, for any predetermined period (e.g., one second or more seconds), Data can be collected without interruption over periods of one minute or more, one hour or more, one day or more, one year or more, etc. This may allow for subsequent editing or synchronization of the acquired video, audio / ultrasound, and / or environmental data for analysis or presentation.

[0108] operating system

[0158] The selection of the operating system depends on the availability of drivers for various sensors. This can be driven by the following: For example, only the Avisoft Ultrasonic microphone driver is compatible with the Windows operating system. However, this selection may have the following implications:

[0109] Inter-process communication: The choice of inter-process communication is influenced by the underlying OS. Similarly, the OS influences the selection of communication between threads. However, development on cross-platform frameworks like Qt can bridge this gap.

[0110] Accessing the system clock: The method for accessing the high-resolution system clock varies depending on the OS, as will be discussed in more detail below. Hardware options

[0159] In certain embodiments, the control system takes the form of a single-board computer. This can be implemented by controller 216. Multiple options are available, such as highly robust military-grade / industrial computers for continuous operation.

[0111] External clock vs. system clock

[0160] Without introducing an external clock into the system, the appropriate clock can be derived from the system clock. Real-time clock values ​​are available. In POSIX systems, the clock_gettime(CLOCK_MONOTONIC, ...) function can return seconds and nanoseconds. The clock resolution can be queried with the clock_getres() function. In embodiments of the control system, the clock resolution should preferably be less than a frame duration of approximately 33 milliseconds. In one embodiment, the system clock is a Unix system.

[0112]

[0161] GetTickCo is used to obtain the number of milliseconds since the system started. The unt64() system function has been developed. The expected resolution of this timer is approximately 10 to 16 milliseconds. While this may serve the same purpose as the clock_gettime() system call, it may be beneficial to check and consider the value wrapping.

[0113]

[0162] On Macintosh computers, the system clock is similarly active. It is possible. The following code snippet has been evaluated and submicron-second resolution has been observed.

[0114] clock_serv_t cclock; mach_timespec_t mts; host_get_clock_service(mach_host_self(),SYSTEM_CLOCK,&cclock); clock_get_time(cclock,&mts);

[0163] In any OS, system calls that return the time must be periodically adjusted. If this occurs, it may move backward. In one embodiment, a monotonically increasing system clock may be employed. GetTickCount64(), clock_gettime(), and clock_get_time() can all satisfy this criterion.

[0115] video file size

[0164] The camera supplier's software automatically divides the images into appropriate sizes. It is unlikely that an output file with a timestamp would be saved. In embodiments of the controller 116, it is desirable to collect video data without interruption, read each frame from the camera 110, and provide the collected video data in a simple form. For example, the controller 116 may be configured to provide the data storage device 120 with video frames of approximately 10 minutes per file in raw format, along with a timestamp header or timestamps between frames. Each file would then be less than 2GB.

[0116] Control system architecture

[0165] Figure 7 is a block diagram showing the components of the acquisition system 700. In this configuration, the acquisition system 700 may be executed by the controller 216. Each block represents a separate process or thread of execution. Controller process

[0166] The control process is configured to start and stop other processes or threads. This can be achieved. Furthermore, the control process may be configured to provide a user interface for the acquisition system 700. The control process may be configured to save activity logs and to record errors that occur during acquisition (for example, in a log). The control process may also be configured to resume any suspended processes or threads.

[0117]

[0167] The method of communication between components may be determined after the selection of the system OS. The user interface for Seth can be either a command-line interface or a graphical interface. The graphical interface can be built on a portable framework such as Qt, which would provide OS independence.

[0118] Video acquisition process

[0168] The video acquisition process communicates directly with camera 210 and timestamped The frame may be configured to be saved to the data storage device 220. The video acquisition process can minimize the possibility of dropped frames by operating with high priority. The video acquisition process can be kept relatively simple by minimizing processing between frames. The video acquisition process can also be configured to ensure proper exposure with the minimum effective shutter speed by controlling the IR illumination emitted by the illumination 212.

[0119] Audio acquisition process

[0169] A separate audio acquisition process uses ultrasonic audio with appropriate timestamps. It may be configured to acquire audio data. In one embodiment, the audio system may comprise an array of microphones 222 arranged in audio communication with the housing 204. In certain embodiments, one or more of the microphones 222 may be positioned within the housing 204. Each microphone in the microphone array may have one or more of the following characteristics: a sampling frequency of approximately 500 kHz, an ADC resolution of approximately 16 bits, a frequency range of approximately 10 kHz to approximately 20 kHz, and an 8th-order, 210 kHz anti-aliasing filter. As an example, each microphone in the microphone array may include a Pettersson M500 microphone (Pettersson Elektronik AB, Uppsala, Sweden) or a functional equivalent. As described above, the audio data captured by the microphones 222 may be time-stamped and provided to a controller 216 for analysis and / or to a data storage device 220 for storage.

[0120] Environmental data acquisition process

[0170] A separate environmental data acquisition process collects environmental data such as temperature, humidity, and light levels. The system may be configured to collect environmental data. Environmental data may be collected at low frequencies (e.g., approximately 0.01 Hz to 0.1 Hz). Environmental data may be stored by the data storage device 220 with a timestamp for each record (e.g., as one or more CSV files).

[0121] Lighting control process

[0171] The lighting control process uses lighting 212 to give the mouse a day-night cycle. It can be configured to control the emitted visible light. In one embodiment, as described above, the camera 210 is configured to remove substantially all visible light and respond only to IR, and since visible light can be filtered so that no IR is generated, this process can avoid affecting video capture.

[0122] Video editing process

[0172] The video editing process involves compressing the acquired video data according to a predetermined set of compressions. It can be configured to repackage the video into a predetermined format. This process can be kept separate from video acquisition to minimize the chance of dropped frames. The video editing process can operate as a low-priority background task or after data acquisition is complete.

[0123] Watchdog process

[0173] The watchdog process monitors the health of the data acquisition process. It can be configured as follows: For example, it may record problems (e.g., in a log) and, if necessary, bring about a restart. The watchdog process can also listen for "pulses" from the components it is monitoring. Generally, a pulse can be a signal sent to the controller 216 that confirms that the components of system 700 are functioning correctly. For example, if a component of system 700 stops functioning, the controller 216 may detect that no pulses are being sent from this component. After this detection, the controller 216 can record the event and issue an alarm. Such alarms may include, but are not limited to, audio alarms and visual alarms (e.g., light, alphanumeric display, etc.). As an alternative to or in addition to such alarms, the controller 216 may attempt to restart the operation of the component, such as by sending a reinitialization signal or switching the power. The method of communication between the components of system 700 and the controller 216 may vary depending on the selection of the OS.

[0124] Mouse marking

[0174] In certain embodiments, mice are marked to facilitate tracking. Obtainable. However, as will be discussed in more detail below, marking may be omitted and tracking may be facilitated by other techniques.

[0125]

[0175] Mouse marking for visual identification involves several non-trivial parameters. Such methods exist. In one embodiment, long-term (several weeks) marking can be performed on mice, minimizing the impact on their communication and behavior by making the markings invisible to the mice themselves. For example, long-term IR-sensitive markers that are invisible within the normal field of vision of mice may be used.

[0126]

[0176] In an alternative embodiment, human hair color and hair bleach are used, The mice's fur can be marked. This method allows for clear identification of the mice over several weeks and can be successfully used in behavioral experiments (e.g., Ohayon et al.). See al. (2013). However, the process of marking the hairs requires anesthesia for the mice, which is an unacceptable process for this mouse monitoring system. Therefore, anesthesia can alter physiological functions, and the hair dye itself can often act as a stimulant that alters the behavior of mice. Since each DO mouse is unique, this can introduce unknown variables as a dye / anesthesia × genotype effect.

[0127]

[0177] Another method employing IR dye-based markers and tattoos is also being used. It can be optimized.

[0178] In another embodiment, a pattern is applied to the back of a mouse as a form of marking. To generate it, shaving may be employed.

[0128] Data Gateway

[0179] During the development phase, a total of less than 2TB of data may be required. The data may include raw and compressed video of samples taken with various cameras and compression methods. Therefore, in addition to integrated USV video data, data transfer of video data for extended periods of 7-10 days during load testing is possible. Video size can be reduced according to the selected compression standard. Sample data storage estimates are given below. test: 1 arena Up to 5 cameras Video duration: Approximately 1-2 hours each Total of approximately 10GB (maximum) Load testing: 1 arena One camera Video duration: 14 days Resolution: Twice the current resolution (960 x 960) Total approximately 2TB production: A total of 120 runs (12-16 arenas, 80 animals per group run, alternating experiments) Duration (each): 7 days Resolution: Twice the current resolution (960 x 960) 32.25TB II. Animal Tracking

[0180] Video tracking of animals such as mice is complex and requires a high level of user involvement. Even in dynamic environments, existing animal monitoring systems cannot be used with genetically heterogeneous animals, making large-scale experiments impossible. As described later, attempting to track numerous different mouse strains in multiple environments using existing systems and methods reveals that these systems and methods are unsuitable for large-scale experimental datasets.

[0129]

[0181] Different colors include black, agouti, albino, gray, brown, nude, and mottled. An exemplary dataset containing mice of various coat colors was used for the analysis. All animals were tested according to the JAX-IACUC procedure outlined below. Mice were tested between 8 and 14 weeks of age. The dataset contained 1857 videos from 59 strains, totaling 1702 hours.

[0130]

[0182] All animals were sourced from Jackson Laboratory's production colonies. In accordance with the certification procedures of the laboratory's Institutional Animal Care and Use Committee guidelines, the behavior of adult mice aged 8-14 weeks was observed. The mice were tested. An open-field behavior assay was performed as described in Kumar (2011). In short, the mice in group housing were weighed and acclimatized to the test room for 30–45 minutes prior to the start of video recording. Hereinafter, data for the first 55 minutes of exercise are presented. Where available, eight males and eight females were tested from each inbred and F1 isogenic line.

[0131]

[0183] In one embodiment, the same open-field device (for example,) is used against a white background. It would be desirable to track multiple animals in Arena 200. Examples of full-frame and cropped video images acquired by the video acquisition system are shown in the first column (full frame) and second column (cropped) of Figure 8A. Examples of ideal tracking frames and actual tracking frames are shown for each environment with various genetic backgrounds (third column (ideal tracking) and fourth column (actual tracking) of Figure 8A).

[0132]

[0184] In another embodiment, one embodiment of the arena 200 includes a feeder and a water dispenser. Video analysis of behavior in harsh environments, such as in the Knockout Mouse Project (KOMP2) at the Jackson Laboratory, would have been desirable (columns 5 and 6 of Figure 8A, respectively).

[0133]

[0185] In the 24-hour setup, mice are kept with a blank bedding and food / water dishes, along with ants. The mice were housed in Arena 200. The mice were confined to Arena 200, and continuous recording was performed under day-night conditions using infrared light emitted by Illuminator 212. The mice moved their bedding and food dishes, and the visible light emitted by Illuminator 212 was altered throughout the day to simulate the day-night cycle.

[0134]

[0186] In the KOMP2 project, data was collected over a five-year period, but the beam As an additional analytical method to detect gait effects that cannot be identified by the break system, it was desirable to perform video-based recording. In gait analysis, the movement of the animal is analyzed. If the animal's gait is abnormal, abnormalities in the skeleton, muscles, and / or nerves may be derived. The KOMP2 project uses a beam break system in which mice are placed in transparent polycarbonate boxes that are fully illuminated with infrared light. The matrix floor is also polycarbonate, and the underlying bench surface is dark gray. Several boxes are positioned at the connection point of two tables to enable joining, and ceiling lighting (e.g., LED lighting) can provide unique high brightness to all boxes.

[0135]

[0187] In one embodiment, a current system using background subtraction and spot detection heuristics is used. Video tracking of this dataset was attempted using Ctrax, a 1970s open-source tracking tool. Ctrax abstracts the mouse frame by frame against five metrics: major and minor axes, the x and y positions of the mouse's center, and the animal's orientation (Branson (2009)). It also utilizes the MOG2 background subtraction model, in which the software estimates both the mean and variance of the video background used for background subtraction. Ctrax fits an ellipse using the shape of the predicted foreground.

[0136]

[0188] In another embodiment, commercially available tracking software that uses its own tracking algorithm Using the LimeLight framework, we attempted to track videos in this dataset. LimeLight performs segmentation and detection using a single keyframe background model. Once a mouse is detected, LimeLight abstracts the mouse relative to its centroid using its own algorithm.

[0137]

[0189] This dataset presents significant challenges for these existing analysis systems. For example, Ctrax and LimeLight combine mouse fur color and environment. Handling the matching proved difficult. Generally, high-contrast environments, such as dark-colored mice on a white background (e.g., black, agouti), yielded good tracking results. However, low-contrast environments, such as light-colored mice on a white background (e.g., albino, gray, or mottled mice), produced unsatisfactory results. Black mice in a white open field achieved high foreground-background contrast, resulting in actual tracking closely matching the ideal. Gray mice visually resemble the arena walls, so their noses are often obscured when they are positioned with their backs to the walls. Albino mice are often not found during tracking because they resemble the background of the arena itself. Mottled mice are often split in two due to their patterned fur color. Although attempts were made to optimize and fine-tune Ctrax for each video, a considerable number of poor tracking frames were still observed, as shown in the actual tracking results in the fourth column (actual tracking) compared to the third column (ideal tracking) in Figure 8A. Discarding bad tracking frames is undesirable because it can lead to biased sampling and distort biological interpretations.

[0138]

[0190] These errors are ideal for tracking in environments such as 24-hour environments and KMOP2 environments. It was observed that the error increased when the target was no longer visible. Furthermore, the error distribution was not random. For example, as shown in the fourth column of Figure 8 (actual tracking), tracking was extremely inaccurate when the mouse was in a corner, near a wall, or on top of the food dish, while tracking was not as inaccurate when the mouse was in the center. In a 24-hour environment, placing a food dish in the arena causes tracking problems if the mouse climbs onto it. Errors also occur in the tracking algorithm in arenas with reflective surfaces, such as KOMP2.

[0139]

[0191] Further investigation into the causes of poor tracking revealed that in most cases, improper tracking is due to the mouse. It was found that this was due to insufficient segmentation from the background. This included cases where mice were removed from the foreground or where the background was included in the foreground due to insufficient contrast. Traditionally, some of these hurdles have been addressed by changing the environment for optimized video data acquisition. For example, to track albino mice, the background color of an open field can be changed to black to increase contrast. However, such environmental changes are not appropriate in this context because environmental color affects the behavior of mice and humans, and such manipulation can potentially confuse experimental results (Valdez (1994), Kulesskaya (2014)). Also, in 24-hour data acquisition systems or KOMP2 arenas, such solutions may not work for mottled mice.

[0140]

[0192] Ctrax uses a single background model algorithm, so other background models A test was conducted to determine if the algorithm could improve tracking results. Twenty-six different segmentation algorithms (Sobral (2013)) were tested, and as shown in Figure 8B, it was found that each of these conventional algorithms worked well under certain circumstances but failed elsewhere. Other available systems and methods for animal tracking rely on background subtraction techniques for tracking. Since all 26 background subtraction methods failed, the results for Ctrax and LimeLight are thought to represent these other techniques. These segmentation algorithms are thought to fail due to improper segmentation.

[0141]

[0193] Thus, many tracking solutions exist for analyzing video data. However, existing solutions have failed to overcome the fundamental problems of proper mouse segmentation and achieve high-quality mouse tracking. Potential confusion arises because no solution adequately addresses the fundamental problems of mouse segmentation and achieves proper segmentation largely through environmental optimization.

[0142]

[0194] Furthermore, the time cost of fine-tuning the parameters of the background subtraction algorithm is exorbitant. This can happen. For example, in 24-hour data tracking, if a mouse sleeps in the same position for an extended period, it becomes part of the background model and becomes untrackable. In typical monitoring, an experienced user would interact with the video for 5 minutes every hour to ensure high-quality tracking results. This level of user interaction is manageable for small, limited experiments, but in large, long-term experiments, prolonged involvement is necessary to monitor tracking performance.

[0143]

[0195] Embodiments of this disclosure overcome these difficulties and enable video data including animals such as mice. We will build a robust next-generation tracker suitable for data analysis. As will be discussed in detail below, an artificial neural network will be employed that achieves high performance under complex and dynamic environmental conditions and does not require continuous fine-tuning by the user, regardless of the genetic characteristics of the fur color.

[0144]

[0196] Convolutional neural networks learn to represent data using multiple levels of abstraction. This is a computational model that includes multiple processing layers to learn. These methods have dramatically improved state-of-the-art speech recognition, visual object recognition, object detection, and many other areas such as drug discovery and genomics (LeCun (2015)). One advantage is that once an efficient network with suitable hyperparameters is developed, the neural network can be easily extended to other tasks simply by adding appropriate training data. Thus, the embodiments disclosed provide a highly generalizable solution for mouse tracking. Neural network architecture

[0197] Three major network architectures have been developed to solve the problem of visual tracking. In one embodiment, as shown in Figure 8C, object tracking may take the form of an elliptical description of a mouse based on a segmentation mask (see Branson (2005)). In alternative embodiments, shapes other than ellipses may be employed.

[0145]

[0198] The elliptic representation is expressed by six variables, also referred to herein as parameters: The position of the animal can be described. In one embodiment, one of the variables may be coordinates that define the position in a predetermined coordinate system (e.g., x and y in a Cartesian coordinate system) representing the pixel position of the mouse in the acquired video frame (e.g., mean center position). That is, a unique pixel position in a plane. Optionally, if necessary, landmarks in the video frame (e.g., corners of the enclosure 204) may be detected to assist in determining the coordinates. In another embodiment, the variables may further include the length of the mouse's major axis and minor axis, as well as the sine and cosine of the vector angle of the major axis. This angle may be defined with respect to the direction of the major axis. The major axis may extend in the coordinate system of the video frame from around the tip of the animal's head (e.g., the nose) to around the end of the animal's body (e.g., around the point where the animal's tail extends from the body). For clarity, in this specification, cropped frames are shown as input to the neural network, while the actual input is an unmarked full frame.

[0146]

[0199] We will use a neural network architecture to determine the elliptic parameters. Exemplary systems and methods are discussed in detail below. It should be understood that other parameters may be used and determined as needed by the embodiments of the disclosure.

[0147]

[0200] In one embodiment, the first architecture is an encoder-decoder-segment This is a maintenance network. As shown in Figure 9, this network predicts a foreground-background segmented image from a given input frame, and the output can be used as a segmentation mask to predict whether or not a mouse is present in terms of pixels.

[0148]

[0201] This first architecture takes a set of features with a small spatial resolution as input (even if It features a feature encoder configured to abstract from 480x480 to 5x5. Many parameters are assigned to the neural network for training. Training can be performed by supervised training, in which case the neural network is presented with examples and produces correct predictions by tuning the parameters. The definition of the final model and all training hyperparameters are listed in Table 3 below.

[0149] [Table 3]

[0150]

[0202] The feature encoder encodes a set of small spatial resolution features in the same form as the original input image. This is followed by a feature decoder configured to return to the original state. In other words, the parameters learned in the neural network reverse the feature coding operation.

[0151]

[0203] Three fully connected layers are added to the encoded features to predict the fundamental direction the ellipse is pointing. A fully connected layer can represent a neural network layer in which each digit of a given layer is multiplied by a different parameter (e.g., a learnable parameter), and the sum of these multiplies produces a single value in a new layer. This feature decoder can be trained to generate foreground-background segmented images.

[0152]

[0204] The first half of the network (encoder) performs batch normalization and ReLU activation. This then utilizes a 2D convolutional layer and a 2D maximum pooling layer. For further details, see Goodfellow (2016).

[0153]

[0205] A starting filter size of 8 was adopted, which doubles after each pooling layer. The kernels used are 5x5 for 2D convolutional layers and 2x2 for max pooling layers. The input video is 480x480x1 (e.g., monochrome), and after repeating these layers 6 times, the resulting shape is 15x15x128 (e.g., 128 colors).

[0154]

[0206] In alternative embodiments, pooling layers of other shapes, such as 3x3, are employed. Obtain. A repeating layer represents a layer with a repeating structure. The neural network learns different parameters for each layer, and each layer is stacked. Six repeating layers were mentioned above, but the number of repeating layers used can be more or fewer.

[0155]

[0207] After another 2D convolutional layer (kernel 5x5, 2x filter) is applied, Different kernel 3x3 and stride 3 2D maximum pools are applied. The 15x15 spatial shape can be further reduced by using a coefficient of 3. The normal maximum pool is kernel 2x2 and stride 2, but each 2x2 grid selects the maximum value and generates one value. These settings select the maximum value in a 3x3 grid.

[0156]

[0208] The final 2D convolutional layer is applied, resulting in a 5x5x512 shape feature bottle. A bottleneck is generated. The feature bottleneck represents the encoded feature set, and the actual matrix values ​​are output by all these matrix operations. The learning algorithm optimizes the encoded feature set so that it is most significant for the task being trained to work well. This feature bottleneck is then passed on to both the segmentation decoder and the angle predictor.

[0157]

[0209] The segmentation decoder uses a stride-transposed 2D convolutional layer. The encoder is reversed, and the pre-downsampling activation is carried over by the summation junction. Note that this decoder does not utilize ReLU activation. The pre-downsampling activation and summation junction may also be called skip connections. From the features of the layer where decoding matches the same shape as the encoder layer, the network can choose between better encoding and state preservation when the encoder is in state.

[0158]

[0210] After the layer returns to a 480x480x8 shape, a separate kernel size of 1x1 is created. The application of convolution results in two monochrome images with depth (background prediction and foreground prediction). The final output is 480×480×2 (two colors). The first color is designated to represent the background. The second color is designated to represent the foreground. For each pixel, the network considers the larger of the two colors as the input pixel. As discussed below, the softmax operation rescales these colors to the cumulative probability that their sum is 1.

[0159]

[0211] After that, softmax is applied to the entire depth. softmax is This is a classification into groups or a form of binmin. Further information on softmax can be found in Goodfellow (2016).

[0160]

[0212] Angle predictions are also generated from the feature bottleneck. This is done using two 2D convolutions. This is achieved by applying batch normalization and ReLU activation to the layer (kernel size 5x5, feature depths 128 and 64). From this, one fully connected layer is flattened and used to generate the shape of four neurons that act to predict the quadrant the mouse head will face. Further details on batch normalization, ReLU activation, and flattening can be found in Goodfellow (2016).

[0161]

[0213] Since the angle is predicted by the segmentation mask, the correct direction (± Only the choice of 180° is required. That is, since an ellipse is predicted, there is only one major axis. One end of the major axis is the direction of the mouse's head. The mouse is assumed to be longer along the head-tail axis. Thus, one direction is +180° (head) and the other direction is -180° (tail). The four possible directions that the encoder-decoder neural network architecture can select are 45-135°, 135-225°, 225-315°, and 315-45° on the polar coordinate grid.

[0162]

[0214] These boundaries were chosen to avoid discontinuities in angle prediction. In particular, as mentioned above, the angle prediction is the prediction of the sine and cosine of the vector angle on the major axis, employing the atan2 function. The atan2 function is discontinuous (at 180°), and the selected boundary avoids these discontinuities.

[0163]

[0215] After the network generates the segmentation mask, Branson( As described in 2009, elliptic fitting algorithms can be applied for tracking. Branson uses weighted sample mean and variance for these calculations, but the segmentation neural network remains invariant to situations representing improvement. For segmentation masks generated by background subtraction algorithms, the projected shading may add errors. The neural network learns to avoid these problems entirely. Furthermore, no significant difference is observed between the use of weighted and unweighted sample mean and variance. The elliptic fitting parameters predicted by weighted and unweighted methods do not differ significantly when using the mask predicted by the disclosed neural network embodiment.

[0164]

[0216] Given a segmentation mask, the sample average of the pixel positions is the center position. It is calculated to represent [this].

[0165]

number

[0166] Similarly, the sample variance of the pixel position is calculated to represent the length of the major axis (a), the length of the minor axis (b), and the angle (θ).

[0167]

number

[0168] To determine the axis length and angle, it is necessary to solve the eigenvalue decomposition equation.

[0169]

number

[0170]

number

[0171]

[0217] The second network architecture is the binning classification network. As shown in Figure 10, the structure of the binning classification network architecture allows for the prediction of a heatmap of the most probable values ​​for each elliptic fitting parameter.

[0172]

[0218] This network architecture abstracts the input image to a small spatial resolution. It begins with a feature encoder. While most regression predictors implement the solution using bounding boxes (e.g., squares or rectangles), the ellipse simply adds one additional parameter: angle. Since the angle is the number of repetitions that are equal at 360° and 0°, the angle parameter is converted into its sine and cosine components. This results in a total of six parameters that are regressiond from the network. The first half of this network encodes a set of features relevant to solving the problem.

[0173]

[0219] Encoded features are transformed from a matrix (array) representing the features into a single vector. The features are flattened by this process. The flattened encoded features are then connected to an additional fully connected layer whose output shape is determined by the desired output resolution (for example, by inputting the feature vector into a fully connected layer). For example, in the case of the X coordinate position of a mouse, there are 480 bins, with one bin for each x column of a 480x480 pixel image.

[0174]

[0220] When the network is active, the maximum value in each heatmap is selected as the most probable value. Each desired output parameter can be implemented as a set of independent, trainable, fully connected layers connected to the encoded features.

[0175]

[0221] Resnet V2 50, Resnet V2 101, Resnet V A wide variety of pre-built feature detectors were tested, including Resnet V2 200, Inception V3, Inception V4, VGG, and Alexnet. A feature detector represents a convolution that operates on the input image. In addition to these pre-built feature detectors, various custom networks were also investigated. This investigation revealed that Resnet V2 200 performed best.

[0176]

[0222] The final architecture is the regression network shown in Figure 11. The regression network then takes an input video frame, extracts features using a Resnet200 CNN, and directly predicts six parameters for elliptic fitting. Each value (the six for elliptic fitting) is continuous and can have an infinite range. The network needs to learn an appropriate range of values. In this way, the numerical values ​​of the ellipse describing the tracked ellipse are predicted directly from the input image. That is, instead of directly predicting the parameters, the regression network does not, but rather selects the most probable values ​​from a set of possible binning values.

[0177]

[0223] Other neural network architectures behave differently. An encoder-decoder neural network architecture outputs the probability that each pixel is a mouse or not. A binning classification neural network architecture outputs bins representing the mouse's position. The class of each parameter is predetermined, and the network (encoder-decoder or binning) only needs to output the probability of each class.

[0178]

[0224] The regression network architecture starts with a feature encoder that abstracts the input into a small spatial resolution. In contrast to the above architecture, regression neural network training relies on a cross-entropy loss function, unlike the mean squared error loss function. Due to memory constraints, only the custom VGG-like network with reduced feature dimensions was tested. The network that worked best was structured with a 2D max pooling layer after two 2D convolutional layers. The kernels used are 3×3 in the case of the 2D convolutional layer and 2×2 in the case of the 2D max pooling layer. The filter depth used first is 16, which is doubled for each 2D max pool layer. This sequence of two convolutional + max pool is repeated 5 times, resulting in a shape of 15×15×256.

[0179]

[0225] This layer is flattened and connected to a fully connected layer for each output. The shape of each output is determined by the desired resolution and range of the prediction. As an example, these encoded features were then flattened and connected to a fully connected layer, resulting in an output shape of 6, which is the number of values the network was required to predict for ellipse fitting. For testing purposes, only the center position was observed and trained on a wide overall image (0 - 480). Additional outputs such as angle prediction can be easily added as additional output vectors. A variety of modern feature encoders were tested, but the data discussed in this document for this network is derived from the 200-layer Resnet V2 that achieved the best results for this architecture (He (2016)). Due to memory constraints, only the custom VGG-like network with reduced feature dimensions was tested. The network that worked best was structured with a 2D max pooling layer after two 2D convolutional layers. The kernels used are 3×3 in the case of the 2D convolutional layer and 2×2 in the case of the 2D max pooling layer. The filter depth used first is 16, which is doubled for each 2D max pool layer. This sequence of two convolutional + max pool is repeated 5 times, resulting in a shape of 15×15×256.

[0180]

[0226] This layer is flattened and connected to a fully connected layer for each output. The shape of each output is determined by the desired resolution and range of the prediction. As an example, these encoded features were then flattened and connected to a fully connected layer, resulting in an output shape of 6, which is the number of values the network was required to predict for ellipse fitting. For testing purposes, only the center position was observed and trained on a wide overall image (0 - 480). Additional outputs such as angle prediction can be easily added as additional output vectors. A variety of modern feature encoders were tested, but the data discussed in this document for this network is derived from the 200-layer Resnet V2 that achieved the best results for this architecture (He (2016)). This layer is flattened and connected to a fully connected layer for each output. The shape of each output is determined by the desired resolution and range of the prediction. As an example, these encoded features were then flattened and connected to a fully connected layer, resulting in an output shape of 6, which is the number of values the network was required to predict for ellipse fitting. For testing purposes, only the center position was observed and trained on a wide overall image (0 - 480). Additional outputs such as angle prediction can be easily added as additional output vectors. A variety of modern feature encoders were tested, but the data discussed in this document for this network is derived from the 200-layer Resnet V2 that achieved the best results for this architecture (He (2016)).

[0181] Training dataset

[0227] To test the network architecture, as described below, OpenCV Using the base labeling interface, a training dataset consisting of 16,234 training images and 568 separate validation images spanning multiple strains and environments was generated. This labeling interface enables fast foreground and background labeling, as well as elliptic fitting, and can be used to immediately generate training data and adapt any network to new experimental conditions through transfer learning.

[0182]

[0228] Interactive watershed-based segmentation The OpenCV library was used to generate contour-based ellipse fittings. Using this software, the user marks points as foreground (e.g., mouse (F)) with a left click and labels other points as background (B) with a right click, as shown in Figure 12A. Keystrokes execute the watershed algorithm, predicting segmentation and ellipses, as shown in Figure 12B. If the user needs to edit the predicted segmentation and ellipses, they simply need to label areas further and run the watershed algorithm again.

[0183]

[0229] The neural network is selected by the user (for example, a researcher). If the prediction falls within a predetermined error tolerance, the user selects the direction of the ellipse. The user makes this selection by choosing one of four basic directions (up, down, left, right). Since the ellipse fitting algorithm selects the precise angle, the user only needs to identify a ±90° range for the direction. Once the direction is selected, all relevant data is saved, and the user is presented with a new frame to label.

[0184]

[0230] The purpose of the labeled dataset is to track good elliptic fitting for mice. The goal was to identify the data. During data labeling, ellipse fitting was optimized so that the center of the ellipse was the mouse's body, with the end of the long axis roughly touching the mouse's nose. The tail was often removed from the segmentation mask to provide a better ellipse fitting.

[0185]

[0231] To train the inference network, three labeled training data are used. A dataset was generated. Each dataset includes a reference frame (input), a segmentation mask, and elliptic fitting. Each training set was generated to track mice in different environments.

[0186]

[0232] The first environment is an array with a constant white background containing 16,802 annotated frames. The setup was Poonfield. The first 16,000 frames were labeled with 65 separate videos taken from one of 24 identical setups. After the network's initial training, it was observed that the network did not function well under specific circumstances not included in the labeled data. Cases of intermediate jumps, anomalous postures, and urination in the arena were typically observed as failures. These failures were identified, correctly labeled, and incorporated into the labeled training set to further generalize and improve performance.

[0187]

[0233] The second environment consists of two different ALPHA-dri bedding and food bowls. The dataset consisted of a standard open field under typical illumination conditions (visible light during the day and infrared light at night). In this dataset, a total of 2,192 frames were labeled across six setups over four days. Of the annotated frames, 916 were acquired from nighttime illumination, and 1,276 were acquired from daytime illumination.

[0188]

[0234] The last labeled dataset is Opto- for the KOMP dataset. The dataset was generated using the M4 open field cage. This dataset contained 1083 labeled frames. All of these labels were sampled across different videos (one frame labeled per video) and eight different setups.

[0189] Neural network training a) Expanding the training dataset

[0235] This training dataset allows you to train by applying reflexes. During training, the network was scaled up eightfold, and the application of small random changes in contrast, brightness, and rotation made it more robust to slight variations in the input data. This scaling is done to prevent the neural network from memorizing the training dataset. If the dataset is memorized, it will not function well with examples (validation) that are not included in the dataset. For further details, see Krizhevsky (2012).

[0190]

[0236] The expansion of the training set is a feature of neural networks since Alexnet. This is an important aspect of training (Krizhevsky (2012)). A handful of training set augmentations are used to achieve good regularization performance. Since the data originates from a bird's-eye view, applying horizontal, vertical, and oblique reflections is easy to instantly increase eightfold on an equivalent training set size. Also, slight rotations and translations are applied to the entire frame at runtime. Rotation augmentation values ​​are sampled from a uniform distribution. Finally, noise, luminance, and contrast augmentations may also be applied to the frame. The random values ​​used for these augmentations are selected from a normal distribution.

[0191] b) Training learning rate and batch size

[0237] The learning rate and batch size of the training were selected independently for each network training. Large-scale networks such as Resnet V2 200 may fall into the memory constraint of the batch size at an input size of 480×480, but good learning rates and batch sizes were experimentally identified using a grid search method. The hyperparameters selected for training these networks are shown in Table 3 above. Model construction, training, and testing were performed in TensorFlow v1.0. The presented training benchmarks were run on the NVIDIA® Tesla® P100 GPU architecture. The hyperparameters were trained through multiple training iterations. After the first training of the network, it was observed that the network was not functioning well under special circumstances where it was under-evaluated in the training data. Cases of intermediate jumps, irregular postures, and urination in the arena were usually observed as unsuccessful. These difficult frames were identified and incorporated into the training dataset to further improve performance. The complete description of the final model definition and all training hyperparameters are listed in Table 3 above.

[0192] Model

[0238] In TensorFlow v1.0, model construction, training, and testing were performed. The presented training benchmarks were run on the NVIDIA(registered trademark) Tesla(registered trademark) P100 GPU architecture.

[0193]

[0239] The hyperparameters were trained through multiple training iterations. After the first training of the network, it was observed that the network was not functioning well under special circumstances where it was under-evaluated in the training data. Cases of intermediate jumps, irregular postures, and urination in the arena were usually observed as unsuccessful. These difficult frames were identified and incorporated into the training dataset to further improve performance. The complete description of the final model definition and all training hyperparameters are listed in Table 3 above. The training and validation loss curve plots shown by three full networks

[0194]

[0240] The training and validation loss curve plots shown by three full networks The results are shown in Figures 13A to 13E. Overall, the training and validation loss curves indicate that all three networks were trained to perform at an average error of 1-2 pixels. Unexpectedly, the binning classification network exhibits an unstable loss curve, indicating overfitting and poor generalization during validation (Figures 13B and 13E). The regression architecture converged to a validation error of 1.2 pixels, which represents better training performance than validation (Figures 13A, 13B, and 13D). However, the Resnet V2 200, the feature extractor that yields the best results, is a large deep network with over 200 layers and 62.7 million parameters, resulting in a substantially long processing time per frame (33.6 ms). Other pre-built general-purpose networks (Zoph (2017)) achieve similar or even lower performance in exchange for shorter computation times. Thus, regression networks are an accurate but computationally expensive solution.

[0195]

[0241] As further shown in Figures 13A, 13B, and 13C, encoder-decode The segmentation architecture converged to a verification error of 0.9 pixels. Not only did the segmentation architecture function well, but the GPU computation efficiency was also good, with an average processing time of 5-6 ms / frame. The video data could be processed at up to 200 fps (6 times real time) on a server-level GPU, the Nvidia® Tesla® P100, and at 125 fps (4.2 times real time) on a consumer-level GPU, the Nvidia® Titan Xp. This high processing speed is thought to be due to the structure having a depth of only 18 layers and only 10.6 million parameters.

[0196]

[0242] Encoder-Decoder Segmentation Network Architecture To identify the relative scale of labeled training data required for good network performance, a benchmark of training set size was also performed. This benchmark was tested by shuffling and random sampling of subsets of the training set (e.g., 10,000, 5,000, 2,500, 1,000, and 500). Each subsampled training set was trained and compared to the same validation set. The results of this benchmark are shown in Figures 14A to 14H.

[0197]

[0243] Generally, the training curves appear indistinguishable (Figure 14A). The training set size does not show any change in performance with respect to the error rate of the training set (Figure 14A). Surprisingly, while the validation performance converges to the same value with more than 2,500 training samples, the error increases with fewer than 1,000 training samples (Figure 14B). Furthermore, as shown, with more than 2,500 training samples, the validation accuracy is better than the training accuracy (Figures 14C-14F), while after matching the training accuracy at 1,000 samples, it begins to show signs of weak generalization (Figure 14G). As indicated by the diverging and increasing validation error rate, using only 500 training samples is clearly overfitting (Figure 14H). This suggests that the training set is no longer large enough for the network to generalize sufficiently. For this reason, good results are obtained only from a network trained with 2,500 labeled images, which takes approximately 3 hours to generate at this labeling interface. Therefore, the exact number of training samples ultimately depends on the difficulty of the visual problem, while the recommended starting point for training samples is around 2,500.

[0198]

[0244] An exemplary video frame showing a mouse being tracked according to the embodiments of the disclosure is Under visible light, the results are shown in Figures 15A and 15B, and under infrared light, they are shown in Figures 15C and 15D. As shown in the figures, the spatial extent of each mouse is color-coded on a pixel-by-pixel basis.

[0199]

[0245] Computational efficiency, accuracy, training stability, and the number of training days required. Given the data, the encoder-decoder segmentation architecture was chosen for predicting the mouse position across the entire video for comparison with other methods.

[0200]

[0246] Inferring the entire video from mice with different coat colors and data collection environments (Figure 8A). Furthermore, the quality of the neural network-based tracking was evaluated by visually assessing the tracking quality. The neural network-based tracking was also compared with the KOMP2 beambreak system, an independent tracking method (Figure 8A, 6th column).

[0201] Experiment Arena a) Open field arena

[0247] One embodiment of Arena 200 was adopted as an open-field arena. The open field arena measures 52cm x 52cm. The floor is made of white PVC plastic, and the walls are made of gray PVC plastic. To aid in cleaning and maintenance, a white 2.54cm panel has been added to all inner edges. Illumination is provided by LED lighting rings (model F&V R300). The lighting rings have been calibrated to produce 600 lux of light in each arena.

[0202] b) Open field arena with 24-hour monitoring

[0248] The open field arena was expanded for testing over several days. Lighting: 212 This was a configuration of ceiling LED lighting set to a standard 12:12 LD cycle. α-Dry was placed in the arena as bedding. A single Diet Gel 76A feeder was placed in the arena to provide food and water. This food source was monitored and replaced when depleted. Each matrix was illuminated at 250 lux during the day and approximately less than 500 lux at night. For nighttime video recording, lighting 212 included IR LED (940 nm) lighting.

[0203] c) KOMP Open Field Arena

[0249] In addition to the custom arena, embodiments of the disclosed systems and methods are commercially available. Benchmarking was also performed on the stem. The Opto-M4 open field cage is constructed using transparent plastic walls. As a result, visual tracking is extremely difficult due to the resulting reflections. The cage measures 42cm x 42cm. The arena is lit by 100-200 lux LED illumination.

[0204] Video acquisition

[0250] All video data is from the video acquisition system discussed in relation to Figures 2 and 7. The data was acquired using one embodiment of the system. Video data was acquired using camera 210 in the form of a Sentech camera (model STC-MB33USB) and a computer lens (model T3Z2910CS-IR) at a resolution of 640 x 480 pixels, 8-bit monochrome depth, and approximately 29 fps (e.g., approximately 29.9 fps). Exposure time and gain were digitally controlled using a target brightness of 190 / 255. The aperture was adjusted to its widest setting so that low analog gain was used to achieve the target brightness. This suppresses the amplification of reference noise. The files were temporarily saved to the local hard drive using the "raw video" codec and "pal8" pixel format. The assay ran for approximately 2 hours and generated approximately 50 GB of raw video files. The ffmpeg software was used overnight to apply a 480x480 pixel crop noise reduction filter and compress the video using an MPEG4 codec (with the quality set to maximum) that generates a compressed video size of approximately 600MB.

[0205]

[0251] To mitigate projection distortion, frame 202 is placed approximately 100 cm above shelf section 202b. Camera 210 was mounted. Zoom and focus were manually set to achieve a zoom of 8 pixels / cm. This resolution minimizes unused pixels on the arena boundary while generating an area of ​​approximately 800 pixels per mouse. Although the KOMP arena is slightly smaller, the same 8 pixels / cm target zoom was utilized.

[0206]

[0252] Using encoder-decoder, segmentation, and neural networks By doing so, 2002 videos (700 hours total) were tracked from the KOMP2 dataset, and the results are shown in Figure 8. These data included 232 knockout lines tested in a 20-minute open-field assay against the C57BL / 6NJ background. Due to the transparency matrix, each KOMP2 arena had a slightly different background, so tracking performance was compared for each of the eight test chambers (n=250 on average (Figure 16)) and for all combined boxes. Across all eight test chambers used by KOMP2, a very high correlation was observed between the two methods in terms of total distance traveled in the open field (R=96.9%). This trend (red arrow) led to two animals being observed with high discrepancy. The video observations showed anomalous postures present in both animals, with one exhibiting a wobbly gait and the other a hunched posture. The wobbly and hunched gaits are thought to result in abnormal beam breaks, leading to an unusually high total distance traveled measure from the beam break system. This example highlights one of the advantages of neural networks that are unaffected by the animal's posture.

[0207]

[0253] Regarding the performance of trained segmentation neural networks However, Ctrax was compared across the entire range of videos and Figure 8A from various test environments. The comparison with Ctrax was motivated by several reasons. In one respect, Ctrax is considered one of the best trackers to date, allowing for fine-tuning of many tracking settings. Also, Ctrax is open source and provides user support. Given the results from the BGS library (Figure 8B), other trackers are expected to perform similarly or below. Twelve animals per group were tracked using both the trained segmentation neural network and Ctrax. The Ctrax settings were fine-tuned for every 72 videos, as described below.

[0208]

[0254] Ctrax includes a variety of settings to optimize tracking capabilities (Branso n(2009)). The authors of this software strongly recommend that the arena be set up under certain criteria to ensure good tracking. In most of the tests discussed herein (e.g., albino mice on a white background), environments are employed that are not designed for Ctrax to function well. Nevertheless, good performance is still achievable with sufficient parameter tuning. Due to the many settings required for operation, Ctrax can easily become time-consuming to achieve good tracking performance. The setup procedure for Ctrax to track mice in the disclosed environment is as follows:

[0209]

[0255] In the first operation, a background model is generated. The core of Ctrax is the background Because it is based on subtraction, a robust background model is functionally essential. The model works best when the mouse is moving. To generate the background model, the video is searched for portions where the mouse is clearly moving, and frames are sampled from those portions. This ensures that the mouse is not included in the background model. This technique significantly improves Ctrax's tracking performance on 24-hour data because the mouse does not move much and is therefore usually incorporated into the background model.

[0210]

[0256] The second step is to set up background subtraction. Here, the standard range is 254. A background brightness normalization method of 0.9 to 255.0 is used. The thresholds applied to isolate mice are adjusted based on preliminary videos, as slight changes in exposure and fur color affect performance. To adjust these thresholds, a set of good starting values ​​is applied, and the videos are scrutinized to ensure generally good performance. In certain embodiments, all videos may be reviewed for cases of mice with their backs to a wall, as these are usually the frames that are most difficult to track due to shading. Morphological filtering may also be applied to remove subtle changes in the environment and to remove the mouse's tail for elliptical fitting. An opening radius of 4 and an occlusion radius of 5 were adopted.

[0211]

[0257] In another operation, the observation results are effectively mouse-like, and can be done with Ctrax. Various tracking parameters are manually adjusted. Considering the time constraints, these parameters were thoroughly adjusted before and after their use with all other tracked mice. If video performance was noticeably insufficient, general settings were fine-tuned to improve performance. For shape parameters, a range based on two standard deviations was determined from the video of individual black mice. The minimum was further lowered because it was anticipated that certain mice would not function well in the segmentation step. This ensures that Ctrax can still find good positions for mice, even though segmentation of the entire mouse is impossible. This technique works well because all setups have the same zoom level 8 and the mice tested are roughly the same shape. The motion settings are very loose in the experimental setup, as only one mouse is tracked in the arena. Under the observation parameters, "Min Area Ignore" is primarily used to remove large detections. Here, detections greater than 2,500 are removed. Under the Hindsight tab, the "Fix Spurious Detections" setting is used to remove detections shorter than 500 frames.

[0212]

[0258] Since Ctrax cannot generate a valid background model, animals are observed continuously for long periods of time. Video from the 24-hour sleep device was manually omitted from the comparison. The cumulative relative error of the total travel distance between Ctrax and the neural network was calculated and is shown (Figure 17A). For each minute of video, the travel distance predictions from both the neural network and Ctrax are compared. This metric measures the accuracy of center of gravity tracking for each mouse. Black, gray, Tracking of mottled mice showed an error rate of less than 4%. However, significantly higher levels of error were observed in albino mice (14%), 24-hour arena mice (27% (orange)), and KOMP2 mice (10% (blue)) (Figure 17A). Therefore, without a neural network tracker, albino tracking, KOMP2, or 24-hour data could not have been properly tracked.

[0213]

[0259] Furthermore, if shading is included in the prediction, the foreground segmentation prediction may not be accurate. In some cases, elliptic fitting was observed to not accurately represent the mouse's posture. In these cases, even when center of gravity tracking was possible, the elliptic fitting itself was highly variable.

[0214]

[0260] JAABA (Kabra (2013)), etc., modern machine learning for action recognition The software utilizes these features for behavioral classification. The variance in elliptic tracking is quantized by the relative standard deviation of the minor axis, as shown in Figure 17B. This metric exhibits the minimum variance across all experimental mice because the width of individual mice remains similar across the wide range of postures represented in behavioral assays when tracking is accurate. Even with small cumulative relative errors in total distance traveled (Figure 17B), high tracking variance was observed in gray and mottled mice (Figure 17A). As expected, high relative standard deviation with respect to the minor axis was observed in the tracking of albino and KOMP2 mice. Thus, the neural network tracker outperforms the conventional tracker in terms of both centroid tracking and elliptic fitting variance.

[0215]

[0261] High-precision encoder-decoder, segmentation, and neural networks. Built as a behavioral tracker, its performance was further tested by two large behavioral datasets. Open-field video data was generated from 1845 mice across 58 strains, including mice of all colors, mottled, nude, and obese varieties (1691 hours). This dataset includes 47 inbred mouse strains and 11 F1 isogenic mouse strains, and is the largest open-field dataset generated according to Bogue's (2018) Mouse Phenome Database.

[0216]

[0262] Figure 18A shows the tracking results regarding total migration distance. Each point represents an individual within the lineage. The boxes represent the mean ± standard deviation. All mice were tracked with high accuracy using a single, untrained network. The majority of mouse strains showed visually verifiable tracking fidelity and excellent performance. The observed motor phenotypes are consistent with publicly available datasets of open-field behavior in mice.

[0217]

[0263] 4 C57BL / 6J mice and 2 BTBR T mice + ltpr3 tf The same neural network was employed to track 24-hour video data collected from / J mice (5th column in Figure 8A). These mice were housed with bedding, food, and water for several days, during which time the food location was changed and the lighting was set to a 12:12 light / dark ratio. Video data was recorded using visible and infrared light sources. The movement of all animals was tracked using the same network under these conditions, and very good performance was observed under light / dark conditions.

[0218]

[0264] The results are shown in Figure 18B, where the eight bright spots and dark spots are, respectively, under different lighting conditions. This represents the dimly lit conditions. As expected, a motor rhythm (curve) accompanied by high levels of spontaneous movement was observed during the dark period.

[0219]

[0265] In summary, video-based tracking of animals in complex environments is useful for understanding animal behavior. This has been a long-standing challenge in the wild (Egnor (2016)). Current state-of-the-art systems do not address the fundamental problems of animal segmentation and rely heavily on visual contrast between foreground and background for accurate tracking. As a result, users need to restrict the environment to achieve optimal results.

[0220]

[0266] In this specification, modern neural networks capable of functioning in complex and dynamic environments This paper describes a neural network-based tracker and its corresponding usage. The use of a trainable neural network addresses fundamental tracking problems (foreground and background segmentation). Testing of three different architectures revealed that an encoder-decoder segmentation network achieves high levels of accuracy and operates at high speed (more than 6 times real time).

[0221]

[0267] By having users label just 2,500 images (approximately 3 hours) In short, a labeling interface is provided that allows for training new networks for specific environments.

[0222]

[0268] The disclosed trained neural network uses two existing solutions. Compared to other systems, it was found to be significantly superior in complex environments. Similar results are expected for any commercially available system utilizing background subtraction techniques. In fact, when 26 different background subtraction methods were tested, each was observed to fail under specific circumstances. However, only one neural network architecture can function for mice of all fur types in multiple environments without the need for fine-tuning or user input. This machine learning technique forms the basis of next-generation tracking architectures for behavioral research, as it enables long-term tracking under dynamic environmental conditions with minimal user input.

[0223]

[0269] One or more embodiments or features of the control systems described herein are digital These can be implemented in electronic circuits, integrated circuits, specially designed application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments or features may include implementations in one or more computer programs that run and / or interpret on a programmable system, which includes a dedicated or general-purpose system and at least one programmable processor that can be coupled to receive and transmit data and instructions to a storage system, at least one input device, and at least one output device. Examples of programmable systems or computer systems include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other.

[0224]

[0270] Programs, software, software applications, applications Computer programs, which may also be called components or code, include machine instructions for a programmable processor and may be implemented in high-level procedural languages, object-oriented programming languages, functional programming languages, logic programming languages, and / or assembly / machine language. In this specification, the term “machine-readable medium” refers to any computer program product, apparatus, and / or device, such as magnetic disks, optical disks, memory, and programmable logic elements (PLDs) used to provide machine instructions and / or data to a programmable processor (including machine-readable media that receive machine instructions as machine-readable signals). "(nal)" represents any signal used to provide machine instructions and / or data to a programmable processor. Machine-readable media can persistently store such machine instructions, such as non-transient solid memory, magnetic hard drives, or any equivalent storage media. Alternatively or in addition, machine-readable media can persistently store such machine instructions, such as processor cache or other random-access memory associated with one or more physical processor cores.

[0225]

[0271] To enable interaction with the user, for example, a cathode that displays information to the user. One or more embodiments or features of the subject matter described herein may be implemented on a computer having a display device such as a video-receiving television (CRT), liquid crystal display (LCD), or light-emitting diode (LED) monitor, as well as a keyboard and pointing device (e.g., mouse, trackball) that allows the user to provide input to the computer. Other types of devices that enable user interaction may also be used. For example, any form of sensory feedback is possible as feedback provided to the user, such as visual feedback, auditory feedback, or tactile feedback, and user input may be accepted in any form, including but not limited to acoustic, speech, or tactile input. Other possible input devices include, but are not limited to, touch-sensitive devices such as touchscreens or single-point or multi-point resistive or capacitive trackpads, speech recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software.

[0226]

[0272] All references cited throughout this application (for example, publication or registration) All references (including patents or equivalents, patent application publications, and non-patent literature or other source materials) are incorporated herein by reference as if each reference were incorporated individually by reference, to the extent that it does not contradict at least partially the disclosures of this application. For example, references that are partially contradictory are incorporated by reference with respect to the partially contradictory portion.

[0227]

[0273] In this specification, when the Markush group or other groups are used, the group in question is the group in question. It is intended that all individual elements of the group, as well as all possible combinations and subcombinations thereof, be individually included in the disclosure.

[0228]

[0274] In this specification, the singular forms "a", "an", and The term "the" implies multiple meanings unless otherwise explicitly indicated in the context. For example, the expression "a cell" includes multiple such cells and their equivalents known to those skilled in the art, and so on. Furthermore, the terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably herein.

[0229]

[0275] In this specification, the term "comprising" means "to possess / possess These terms are synonymous with "(including)," "having," "containing," and "characterized by," and can be used interchangeably. Each of these terms is further broader or more open-ended, and does not exclude additional elements or method steps that are not listed.

[0230]

[0276] In this specification, the term "consisting of" means Any elements, steps, or components not specified in the claims are excluded.

[0231]

[0277] In this specification, the term "essentially consisting of" The phrase "sentially of" does not exclude any elements or steps that do not substantially affect the fundamental and novel characteristics of the claims. In no event herein may the terms "comprising," "consisting essentially of," and "consisting of" be replaced by any of the other two terms.

[0232]

[0278] The embodiments described herein as examples are not specifically disclosed herein. This can be suitably realized without one or more elements being omitted, or without any or all limitations.

[0233]

[0279] Expression: "The claim described in any one of claims XX to YY (of any of cl The expression "aims XX-YY)" (where XX and YY represent claim numbers) is intended to provide alternative forms of multiple dependent claims and, in some embodiments, may be used interchangeably with the expression "as in any one of claims XX-YY".

[0234]

[0280] Unless otherwise specified, all technical terms and Scientific terms have the same meanings as those commonly understood by those skilled in the art in the field to which the embodiments of the disclosure belong.

[0235]

[0281] In this specification, for example, temperature range, time range, composition range, or concentration range Whenever a range is given, it is intended that all intermediate and subranges, as well as all individual values ​​within the given range, are included in this disclosure. In this specification, a range specifically includes the values ​​provided as endpoint values ​​of that range. For example, a range of 1 to 100 specifically includes the endpoint values ​​of 1 and 100. It is understood that any subranges or individual values ​​within a range or subrange included in this specification may be excluded from the claims.

[0236]

[0282] In the foregoing and in the claims, "at least one of the following (at l Expressions such as "at least one of" or "one or more of" may be followed by a conjugated list of elements or features. The term "and / or" may also appear as a list of two or more elements or features. Unless there is a clear implicit or explicit inconsistency in the context, such expressions are intended to mean either any of the elements or features in the list individually, or any combination of any of the enumerated elements or features with any of the other enumerated elements or features. For example, the expressions "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or a combination of A and B," respectively. The same interpretation is intended for lists containing three or more items. For example, the expression "at least one of A, B, and C" "one of A, B, and C)", "one or more of A, B, and C", and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C," respectively. Furthermore, in the foregoing and in the claims, the use of the term "based on" is intended to mean "based at least in part on," so as to allow for features or elements not listed.

[0237]

[0283] The terms and expressions used herein are for illustrative purposes only. This is not limiting in any way, and the use of such terms and expressions is not intended to exclude any equivalents of the illustrated and described features or any part thereof, while it is recognized that various modifications are possible within the scope of the claimed embodiments. Therefore, although this application may include descriptions of preferred embodiments, exemplary embodiments, and optional features, it is understood that modifications and variations of the concepts disclosed herein can be made by those skilled in the art. Such modifications and variations are considered to be within the scope of the disclosed embodiments as defined by the appended claims. The specific embodiments described herein are examples of useful embodiments of this disclosure and can, of course, be performed by those skilled in the art using many variations of the devices, device components, and method steps described herein. As will be obvious to those skilled in the art, the methods and devices useful for these methods may include many optional configurations, processing elements, and steps.

[0238]

[0284] The embodiments described herein do not deviate from the spirit or essential characteristics thereof, and are not related to other This can be embodied in specific forms. Therefore, the above embodiments should be considered illustrative rather than limiting in any respect to the subject matter described herein.

[0239] References

[0285] The following references all contain information that is referenced in this specification. It will be incorporated into the book.

[0240] [ka]

[0241] [ka]

[0242] [ka]

Claims

1. A method of tracking animals, The processor receives video data representing the observation of an animal, The aforementioned processor, Receiving input video frames extracted from the aforementioned video data, Based on the input video frame, an elliptic description of at least one animal is generated, the elliptic description is defined by predetermined elliptic parameters, and To provide data for at least one of the animals, including values ​​that characterize the predetermined elliptic parameters, The steps include: running a neural network architecture configured to perform the following; Methods that include...

2. The method according to claim 1, wherein the elliptic parameter is a coordinate representing the position of the animal in a plane, the length of the major axis and the length of the minor axis of the animal, and the angle to which the animal's head is facing, defined with respect to the direction of the major axis.

3. The aforementioned neural network architecture, Predicting a foreground-background segmented image from an input video frame. In terms of pixels, predict whether an animal is present in the input video frame based on the segmented image. Based on the prediction from the perspective of the aforementioned pixels, output a segmentation mask, and To fit the portion of the segmentation mask where the animal is predicted to exist to an ellipse, and to determine the values ​​that characterize the predetermined ellipse parameters, The method according to claim 1, wherein the encoder-decoder segmentation network is configured to perform the following.

4. The encoder-decoder-segmentation network, A feature encoder configured to abstract the aforementioned input video frame into a set of features with a small spatial resolution, A feature decoder configured to convert the aforementioned set of features into the same shape as the input video frame and output the foreground-background segmented image, An angle predictor configured to predict the angle at which the animal's head is facing, The method according to claim 3, comprising:

5. The method according to claim 1, wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of the most probable values ​​of each elliptic parameter in the elliptic description.

6. The method according to claim 5, wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, and the abstraction is used to generate the heatmap.

7. The method according to claim 1, wherein the neural network architecture comprises a regression network configured to extract features from input video frames and directly predict values ​​that characterize each of the elliptic parameters.

8. The method according to claim 1, wherein the animal is a rodent.

9. It is an animal tracking system, A data storage device that maintains video data representing animal observations, The system comprises a processor configured to receive video data from the data storage device and to implement a neural network architecture, wherein the neural network architecture is Receiving input video frames extracted from the aforementioned video data, Based on the video frames, an elliptic description of at least one animal is generated, the elliptic description is defined by predetermined elliptic parameters, and To provide data for at least one of the animals, including values ​​that characterize the predetermined elliptic parameters, A system configured to perform the following actions.

10. The system according to claim 9, wherein the elliptic parameter is a coordinate representing the position of the animal in a plane, the length of the major axis and the length of the minor axis of the animal, and the angle to which the animal's head is facing, defined with respect to the direction of the major axis.

11. The aforementioned neural network architecture, Predicting a foreground-background segmented image from an input video frame. In terms of pixels, predict whether an animal is present in the input video frame based on the segmented image. Based on the prediction from the perspective of the aforementioned pixels, output a segmentation mask, and To fit the portion of the segmentation mask where the animal is predicted to exist to an ellipse, and to determine the values ​​that characterize the predetermined ellipse parameters, The system according to claim 9, which is an encoder-decoder-segmentation network configured to perform the following.

12. The encoder-decoder-segmentation network, A feature encoder configured to abstract the aforementioned input video frame into a set of features with a small spatial resolution, A feature decoder configured to convert the aforementioned set of features into the same shape as the input video frame and output the foreground-background segmented image, An angle predictor configured to predict the angle at which the animal's head is facing, The system according to claim 11, comprising:

13. The system according to claim 9, wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of the most probable values ​​of each elliptic parameter in the elliptic description.

14. The system according to claim 13, wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a small spatial resolution, and the abstraction is used to generate the heatmap.

15. The system according to claim 9, wherein the neural network architecture comprises a regression network configured to extract features from input video frames and directly predict values ​​characterizing each of the elliptic parameters.

16. The system according to claim 9, wherein the animal is a rodent.

17. A non-temporary computer program product that stores instructions, wherein the instructions, when executed by at least one data processor of at least one computing system, The steps include receiving video data representing animal observations, Steps to implement a neural network architecture, The method includes the following, and the neural network architecture is: Receiving input video frames extracted from the aforementioned video data, Based on the input video frames, an elliptic description of at least one animal is generated, the elliptic description is defined by predetermined elliptic parameters, and To provide data for at least one of the animals, including values ​​that characterize the predetermined elliptic parameters, A non-temporary computer program product configured to perform the following actions.

18. The computer program product according to claim 17, wherein the elliptic parameter is a coordinate representing the position of the animal in a plane, the length of the major axis and the length of the minor axis of the animal, and the angle to which the animal's head is facing, defined with respect to the direction of the major axis.

19. The aforementioned neural network architecture, Predicting a foreground-background segmented image from an input video frame. In terms of pixels, predict whether an animal is present in the input video frame based on the segmented image. Based on the prediction from the perspective of the aforementioned pixels, output a segmentation mask, and To fit the portion of the segmentation mask where the animal is predicted to exist to an ellipse, and to determine the values ​​that characterize the predetermined ellipse parameters, The computer program product according to claim 17, which is an encoder-decoder-segmentation network configured to perform the following.

20. The encoder-decoder-segmentation network, A feature encoder configured to abstract the aforementioned input video frame into a set of features with a small spatial resolution, A feature decoder configured to convert the aforementioned set of features into the same shape as the input video frame and output the foreground-background segmented image, An angle predictor configured to predict the angle at which the animal's head is facing, A computer program product according to claim 19, comprising:

21. The method according to claim 17, wherein the neural network architecture comprises a binning classification network configured to predict a heatmap of the most probable values ​​for each elliptic parameter of the elliptic description.

22. The method according to claim 21, wherein the binning classification network comprises a feature encoder configured to abstract the input video frames to a smaller spatial resolution, and the abstraction is used to generate the heatmap.

23. The method according to claim 17, wherein the neural network architecture comprises a regression network configured to extract features from input video frames and directly predict values ​​that characterize each of the elliptic parameters.

24. The method according to claim 17, wherein the animal is a rodent.

25. The following is the best system: It's an arena, Frame, A housing attached to the frame and dimensionally defined to house an animal, the housing including a door configured to allow access to the interior of the housing, Arenas including; and It is an acquisition system, camera; At least two sets of light sources, each set of light sources configured to emit light incident on the housing at different wavelengths from each other, At least two sets of light sources configured to acquire video data of at least a portion of the housing when the camera is illuminated by at least one of the multiple sets of light sources; The camera and the multiple sets of light sources are electrically connected, The camera acquires video data and generates control signals that control the emission of light from the multiple sets of light sources, and Receiving video data acquired by the aforementioned camera, A controller configured to perform the following actions; and A data storage device electrically connected to the controller, configured to store video data received from the controller. An acquisition system that includes this.

26. The system according to claim 25, wherein at least a portion of the housing is substantially opaque to visible light.

27. The system according to claim 25, wherein at least a portion of the housing is formed of a material that is substantially opaque with respect to visible light wavelengths.

28. The system according to claim 25, wherein at least a portion of the housing is formed of a material that is substantially inreflective to infrared light wavelengths.

29. The system according to claim 25, wherein at least a portion of the housing is formed of a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

30. The system according to claim 25, wherein a first set of light sources comprises one or more first illuminators configured to emit light at one or more visible light wavelengths, and a second set of light sources comprises one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

31. The system according to claim 30, wherein the wavelength of the infrared light is approximately 940 nm.

32. The system according to claim 25, wherein the camera is configured to acquire video data with a resolution of at least 480 x 480 pixels.

33. The system according to claim 25, wherein the camera is configured to acquire video data at a frame rate higher than the frequency of mouse movements.

34. The system according to claim 25, wherein the camera is configured to acquire video data at a frame rate of at least 29 frames per second (fps).

35. The system according to claim 25, wherein the camera is configured to acquire video data having at least 8 bits of depth.

36. The system according to claim 25, wherein the camera is configured to acquire video data at infrared wavelengths.

37. The system according to claim 25, wherein the controller is configured to compress video data received from the camera.

38. The system according to claim 37, wherein the controller is configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing distributed-based background subtraction.

39. The system according to claim 38, wherein the filter of the MPEG codec is Q0 HQDN3D.

40. The system according to claim 30, wherein the controller is configured to request the first light source to illuminate the housing according to a schedule that simulates a light-dark cycle.

41. The system according to claim 30, wherein the controller is configured to request the first light source to illuminate the housing with visible light having an intensity of approximately 50 lux to approximately 800 lux during the bright portion of the light-dark cycle.

42. The system according to claim 30, wherein the controller is configured to request the second light source to irradiate the housing with infrared light such that the temperature rise of the housing due to infrared irradiation is less than 5°C.

43. The system according to claim 30, wherein the controller is configured to request the first light source to illuminate the housing according to 1024 levels of illumination scaled logarithmically.

44. A step of illuminating a housing configured to contain an animal with at least one set of light sources, wherein each set of light sources is configured to emit light of different wavelengths from each other; The steps include: acquiring video data of at least a portion of the housing illuminated by at least one of the multiple sets of light sources using a camera; A controller electrically connected to the camera and the multiple sets of light sources generates a control signal that operates to control the acquisition of video data by the camera and the emission of light by the multiple sets of light sources. The controller receives video data acquired by the camera, A method that includes this.

45. The method according to claim 44, wherein at least a portion of the housing is substantially opaque to visible light.

46. The method according to claim 44, wherein at least a portion of the housing is formed of a material that is substantially opaque to visible light wavelengths.

47. At least a portion of the housing is formed of a material that is substantially inreflective to infrared light wavelengths, claim The method described in paragraph 44.

48. The method according to claim 44, wherein at least a portion of the housing is formed of a sheet of polyvinyl chloride (PVC) or polyoxymethylene (POM).

49. The method according to claim 44, wherein a first set of light sources comprises one or more first illuminators configured to emit light at one or more visible light wavelengths, and a second set of light sources comprises one or more second illuminators configured to emit light at one or more infrared (IR) light wavelengths.

50. The method according to claim 49, wherein the wavelength of the infrared light is approximately 940 nm.

51. The method according to claim 44, wherein the camera is configured to acquire video data with a resolution of at least 480 x 480 pixels.

52. The method according to claim 44, wherein the camera is configured to acquire video data at a frame rate higher than the frequency of mouse movements.

53. The method according to claim 44, wherein the camera is configured to acquire video data at a frame rate of at least 29 frames per second (fps).

54. The method according to claim 44, wherein the camera is configured to acquire video data having at least 8 bits of depth.

55. The method according to claim 44, wherein the camera is configured to acquire video data at infrared wavelengths.

56. The method according to claim 44, wherein the controller is configured to compress video data received from the camera.

57. The method according to claim 56, wherein the controller is configured to compress video data received from the camera using an MPEG4 codec that includes a filter employing distributed-based background subtraction.

58. The method according to claim 57, wherein the filter of the MPEG codec is Q0 HQDN3D.

59. The method according to claim 49, wherein the controller is configured to request the first light source to illuminate the housing according to a schedule that simulates a light-dark cycle.

60. The method according to claim 49, wherein the controller is configured to request the first light source to illuminate the housing with visible light having an intensity of approximately 50 lux to approximately 800 lux during the bright portion of the light-dark cycle.

61. The method according to claim 49, wherein the controller is configured to request the second light source to irradiate the housing with infrared light such that the temperature rise of the housing due to infrared irradiation is less than 5°C.

62. The controller is configured to request the first light source to illuminate the housing according to 1024 levels of illumination scaled logarithmically, as per claim 49. Law.

Images