Information processing device, information processing method, and program

Abstract

The present invention addresses the problem of identifying each individual among mice from an image obtained by imaging one or more mice acting in a certain activity range.　An image processing device 2 comprises an image acquisition unit 51, a part extraction unit 52, and an individual identification unit 54. The image acquisition unit 51 acquires a moving image obtained as a result of imaging states in which one or more animals act within the activity range. An outline detection unit 57 detects the outline of the body of each of the one or more animals for each of a plurality of unit images included in the moving image. The individual identification unit 54 analyzes, in every time series, a plurality of outlines detected from each of the plurality of unit images by the outline detection unit 57, and identifies, on the basis of the analysis result, each individual among the one or more animals in each of the plurality of unit images. Thereby, the problem is addressed.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] As a conventional technique, there is a technique for processing biological information of an individual while identifying an individual of an animal such as a cat (for example, Patent Document 1 etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of the conventional technique of the above document, although biological data of an animal can be measured non-contact, it does not cope with individual identification in the presence of always moving animals or a plurality of animals. In addition, the conventional technique requires a thermograph etc. in addition to a camera in order to confirm the health state of the identified animal in processing the biological information of the individual, so it is costly. Furthermore, the above document does not mention applications other than health state management.

[0005] The present invention has been made in view of such a situation, and an object thereof is to enable discrimination of individuals of one or more mice that are active within a certain range of movement.

Means for Solving the Problems

[0006] To achieve the above object, an information processing apparatus according to an aspect of the present invention image acquisition means for acquiring an image to be analyzed configured by arranging a plurality of unit images in the time direction, which is obtained as a result of imaging the state in which one or more animals are active within a certain range of movement; For each of the plurality of unit images, using a skeleton estimation model that estimates and outputs the skeleton of an animal's body when a unit image is input, an outline detection means for detecting the outline of each body of the one or more animals; Based on the output obtained by inputting the time series of the outline of each body of the one or more animals detected from each of the plurality of unit images by the outline detection means into an individual identification model that outputs an individual of the animal when the time series of one or more outlines of the animal's body is input, an individual identification means for identifying the individual of each of the one or more animals in each of the plurality of unit images; A designation means for designating an analysis attribute for the image to be analyzed; Based on the analysis attribute of the image designated by the designation means, a model selection means for selecting a target to be applied to the outline detection means from among a plurality of types of the skeleton estimation models and a target to be applied to the individual identification means from among a plurality of types of the individual identification models; It is provided with. In this way, a plurality of types of skeleton estimation models and a plurality of types of individual identification models are prepared, and based on the analysis attribute of the image designated by the designation means, a skeleton estimation model as a target to be applied to the outline detection means is selected from among the plurality of types of skeleton estimation models, and an individual identification model as a target to be applied to the individual identification means is selected from among the plurality of types of individual identification models. Then, according to an analysis instruction, for each of the plurality of unit images included in the image, the outline of each body of one or more animals is detected, the detected outline is analyzed for each time series, and based on the analysis result, the individual of each of the one or more animals is identified, so that the individual of the animal can be distinguished from an image capturing one or more animals acting within a certain activity range.

[0007] An information processing method and a program corresponding to the above information processing apparatus according to an aspect of the present invention are also provided as an information processing method and a program according to an aspect of the present invention.

Effect of the Invention

[0008] According to the present invention, it is possible to distinguish one or more individual mice that move within a certain range of movement.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing the configuration of an information processing system including an image processing apparatus according to an embodiment of the information processing apparatus of the present invention.

[0011] (First Embodiment) The information processing system shown in FIG. 1 includes a camera 1 installed (arranged) to image a cage C containing one or more mice (mouse X1 and X2 in the example of FIG. 1) from above, and an image processing device 2 connected to the camera 1 via a network N. The network N includes a wired network, a wireless network, and the like. The cage C is a housing means for allowing an animal to move within a certain range of movement.

[0012] The camera 1 is, for example, a digital camera or a network camera that captures a video, and outputs a video captured from above inside the cage C to the image processing device 2. Here, a video refers to an image composed of a plurality of unit images arranged in the time direction, and is also called a video. Although a field image may be adopted as the unit image, a frame image is adopted here.

[0013] The image processing device 2 uses the video acquired from the camera 1 and the learning model stored in the model DB42 (details will be described later) to identify the individuals of the plurality of mice X1 and X2 included as subjects in the video. The image processing device 2 further detects elements (causes) that form the basis of behaviors such as the sociality of each of the mice X1 and X2, the interaction between the mice X1 and X2 existing in the same range of movement, and the relationship between the mice X1 and X2 from the behavior patterns (habits, etc.) of the mice X1 and X2 respectively identified as individuals. The image processing device 2 generates a tracking image with at least one of markers such as parts (tracking points) of the body skeletons of one or more mice included in the frame image extracted from the video and individual IDs (Identifications), and the outlines (contours with the surroundings) of the respective mice, and outputs the tracking image to an output unit 16 such as a display. If a printer is connected to the image processing device 2 as the output unit 16, the tracking image can also be printed. The object is a contour line or a mask image that shows the outline of the mouse individual. The marker includes, for example, a tracking point indicated by a figure such as a circle, a triangle, a square, or an individual ID indicated by alphanumeric characters, etc. Regarding the functional configuration and details of the processing of the image processing apparatus 2, they will be described later with reference to the drawings after FIG. 3.

[0014] FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the information processing apparatus of the present invention among the information processing systems of FIG. 1.

[0015] The image processing apparatus 2 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0016] The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 18 to the RAM 13. In the RAM 13, data and the like necessary for the CPU 11 to execute various processes are also appropriately stored.

[0017] The CPU 11, the ROM 12, and the RAM 13 are interconnected via the bus 14. The input / output interface 15 is also connected to this bus 14. The output unit 16, the input unit 17, the storage unit 18, the communication unit 19, and the drive 20 are connected to the input / output interface 15. The output unit 16 is composed of a display, a speaker, etc., and outputs images and sounds. The input unit 17 is composed of a keyboard, a mouse, etc., and inputs various information in response to a user's instruction operation. The storage unit 18 is composed of a hard disk, etc., and stores data of various information.

[0018] The communication unit 19 controls communication with other communication targets (for example, the camera 1 in FIG. 1) via the network N. A removable medium 21 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 20. The program read from the removable medium 21 by the drive 20 is installed in the storage unit 18 as necessary. Further, the removable medium 21 can also store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0019] FIG. 3 is a functional block diagram showing a first embodiment of the information processing system in FIG. 1, that is, a first embodiment of the functional configuration of the image processing apparatus 2 in FIG. 2.

[0020] In the storage unit 18 of the image processing apparatus 2 shown in FIG. 2, an image DB 41, a model DB 42, and a material DB 43 are stored.

[0021] The image DB 41 stores data of a moving image acquired from the camera 1, data of a plurality of frame images (still image data) constituting the moving image, data of a tracking image in which an object or a marker indicating an individual for animal tracking is added to the frame image, and a table (data frame) showing the transition of the position information of the parts of the mice X1 and X2.

[0022] The model DB 42 stores a plurality of learning models. Specifically, the model DB 42 stores a plurality of types (one or more) of skeleton estimation models for estimating the respective skeletons of each animal (mouse, rat, etc.), a plurality of types (one or more) of individual identification models for identifying each individual of an animal (if there are two of the same animal, each animal is identified as a different individual), a plurality of types (one or more) of mathematical models for determining the behavior of each individual of an animal, and the like. The skeleton estimation model is a learning model configured to output the skeleton of an animal when an image of the animal is input. The individual identification model is a learning model configured to output the individual of the animal when the time series of one or more parts of the animal's body is input. The mathematical model is a model for analyzing animal behavior that is created to output information indicating what kind of behavior an animal has when one or more images of the animal that change over time are input. As a result of machine learning and updating being performed on videos and still images for each individual animal prepared in advance, when a new video or still image is input, learned learning data for identifying the animal, identifying the individual, and further analyzing the behavior of each individual from the input image and outputting the analysis results is stored. Such learned learning data is referred to as the "learning model" in this specification.

[0023] Specifically, from a video obtained by photographing the activities of a mouse as an example of an animal, a pre-selected skeletal estimation model for mice estimates the arrangement (skeleton) of the bones of one or more mouse heads, necks, arms, legs, tails, etc., and the position and movement of feature points (body parts such as the eyes, nose, ears, and toes of the mouse) extracted from the skeleton of the animal are used by a pre-selected individual identification model for mice to identify the individual, and the mathematical model outputs the result of determining the behavior of that individual. That is, the learning model in this first embodiment is a learning model that analyzes the individual and the behavior of the individual of the animal using the techniques of skeletal estimation, individual identification, and behavior estimation. Among the above model groups, for example, inside the skeletal estimation model, data of any pixel in the image (all pixels in a model using CNN) is input, and an appropriate value (such as the coordinates of the position of the nose or the probability of the coordinates of the nose) is calculated according to the model for each animal. Note that CNN refers to a neural network mainly composed of convolutional layers and pooling layers.

[0024] The learning model stored in the model DB42 is a reference learning model generated by a learning device (not shown) performing machine learning using one or more (a large number) of learning data in advance, and newly learned data is also added.

[0025] For machine learning, for example, a convolutional learning type neural network or the like can be applied. Note that the convolutional learning type neural network is only an example, and other machine learning methods may be applied. Furthermore, the learning model is not limited to the machine learning model, and a discriminator that discriminates an animal individual by a predetermined algorithm may be adopted. That is, the learning model may be one that is learned and generated so that when an image is input, it outputs by assigning an attribute when the skeleton, outline, and movement of each mouse included in the image match a mouse with known movement.

[0026] The material DB43 stores data that serves as a material for obtaining the reasons for mouse behavior, sociality, etc. identified based on the learning model. The data in the material DB43 is data that associates mouse behavior with sociality derived from the behavior and the relationship with other mice, and can be used for determining various habits, quirks, and ecology of mouse behavior. For example, not only simple mouse movements, but also data that can derive habits such as mice that make such movements having such sociality and having such a relationship with other mice. The material DB43 stores data that associates the conditions under which a mouse behaves with the mouse behavior pattern derived from the conditions. That is, the material DB43 stores data that serves as a judgment material for determining what kind of behavior the behavior of the mouse detected from the image by the mathematical model is. As the data serving as the judgment material, for example, behavior data such as exploratory behavior, feeding and drinking, running and walking, sleeping, scratching, grooming, and fighting are stored.

[0027] As shown in FIG. 3, the image processing apparatus 2 includes an image acquisition unit 51, a part extraction unit 52, a data frame generation unit 53, an individual identification unit 54, a marker image generation unit 55, an action determination unit 56, and the like.

[0028] The image acquisition unit 51 acquires a video obtained as a result of imaging the activities of one or more animals such as mice X1, X2, etc. within a certain action range.

[0029] The image acquisition unit 51 includes a video acquisition unit 61 and a unit image generation unit 62. The video acquisition unit 61 acquires the video (moving image) captured by the camera 1. The unit image generation unit 62 generates an image to be analyzed, which is composed of a plurality of unit images arranged in the time direction, from the video (moving image) acquired by the video acquisition unit 61, and stores it in the image DB 41. That is, the unit image generation unit 62 generates a group (frame image) of a plurality of unit images (still images) in frame units from the video.

[0030] The body part extraction unit 52 extracts the body parts of one or more animals for each of the plurality of unit images using the selected skeleton estimation model from among the plurality of types of skeleton estimation models in the model DB 42. That is, the body part extraction unit 52 sequentially reads out the plurality of frame images in the image DB 41, and recognizes the image area of the animal that moves itself from the unit images included in each frame image. That is, the background part of the frame image and the contour part of the animal's body are separated to recognize the image area of the animal's body. The parts of a part of the body are, for example, one or more feature points among the left and right eyes, nose, left and right ears, tips of the left and right front legs, tips of the left and right hind legs, tip of the tail, joints of bones and joints, and the center of gravity of the body. The unit image includes, in addition to the above-described frame image, one pixel (picture element), a group of a plurality of pixels, and the like.

[0031] The body part extraction unit 52 includes an individual recognition unit 71 and a body part detection unit 72. The individual recognition unit 71 recognizes, in each of the plurality of unit images included in the frame image, those with changes (movements) as part of one acting individual. Specifically, the individual recognition unit 71 binarizes the unit image and recognizes the color different from the background image as part of the acting individual. The individual recognition unit 71 sets the portion where the color changes as the boundary of the region by comparing with the frame images before and after in time series. For each of the plurality of unit images, the body part detection unit 72 superimposes the skeleton output from the skeleton estimation model on the region of what has been recognized as a part of the moving individual, extracts the body part of the animal, and uses that part as a tracking point. At least one body part such as the nose and the left and right eyes is detected. In this embodiment, the body part extraction unit 52 including the individual recognition unit 71 and the body part detection unit 72 and the skeleton estimation model are combined to extract the body parts of one or more animals from the image. However, a body part extraction model that outputs the body parts of animals when a unit image is input may be stored in advance in the model DB 42, and the body parts of the animals may be extracted by inputting the unit image into the model DB 42.

[0032] The data frame generation unit 53 forms the body parts extracted by the body part extraction unit 52 into a data frame. The data frame generation unit 53 specifies, for each region of each animal, the position of a specific part (such as eyes, nose, ears, feet, tail, skeleton, etc.) of the animal in coordinates indicating the distance from a predetermined reference point in the image. Specifically, the data frame generation unit 53 generates a table that associates the body parts extracted by the body part extraction unit 52 with the position information represented in a two-dimensional (planar) coordinate system (x-axis, y-axis, etc.) with a certain reference point of the frame image or the cage C included in a part of the moving individual as a reference. The generated table is called a data frame. The x-axis refers to the horizontal axis of the two-dimensional (planar) coordinate system, and the y-axis refers to the vertical axis of the plane. That is, the data frame generation unit 53 generates a data frame indicating the transition of the position of the part that changes as the animal moves. When acquiring the spatial position as a tracking point, a three-dimensional (stereoscopic) coordinate system (x-axis, y-axis, z-axis) is used. The z-axis refers to the depth axis of the three-dimensional (stereoscopic) coordinate system.

[0033] The individual identification unit 54 identifies the individuals of each of the one or more animals in each of the plurality of unit images based on the output obtained by inputting the time series of the body parts of each of the one or more animals extracted from each of the plurality of unit images by the part extraction unit 52 to the selected individual identification model among the plurality of types of individual identification models in the model DB 42. Specifically, the individual identification unit 54 analyzes, for each time series, the one or more parts extracted from each of the plurality of unit images by the part extraction unit 52 using the individual identification model selected for images from among the plurality of types, and identifies the individuals of each of the one or more mice X1, X2 in each of the plurality of unit images based on the analysis result. The individual identification unit 54 analyzes how the position coordinates of the parts change over time, and identifies (classifies as different individuals) each of the one or more animals included in the frame image based on the analysis result. That is, the individual identification unit 54 identifies what kind of animal part the one or more parts are, and which individual is the animal having that part.

[0034] Specifically, the individual identification unit 54 refers to the model DB 42, analyzes the data frame generated by the data frame generation unit 53 for each time series, identifies the individuals of each of the one or more mice X1, X2 in each of the plurality of data frames based on the analysis result, and assigns the individual IDs of the identified mice X1, X2 to the tracking points (tracking points).

[0035] The individual identification unit 54, for example, converts into data the positional relationship of each part of each individual at an arbitrary point in time and the frequency of satisfying a certain condition. Here, the certain condition is, for example, when the positions of a certain part and other parts exist at the positions (predetermined range) of predetermined coordinates for a certain period, or when the state where the distance from the parts of other individuals is zero or approximates zero continues for a certain period.

[0036] The individual identification unit 54 instructs the marker image generation unit 55 to generate a marked image (tracking image) obtained by superimposing at least one of a marker indicating the tracking points (tracking points) of the mice X1 and X2 identified by the individual identification unit 54 and an object for visually identifying the individuals of the mice X1 and X2 on the unit image, and outputs the marked image to the output unit 16.

[0037] The marker image generation unit 55 generates a marked image in which the body parts of the mice X1 and X2 extracted by the part extraction unit 52 and the markers indicating the individuals of the mice X1 and X2 identified by the individual identification unit 54 are associated with the parts. Specifically, the marker image generation unit 55 generates a marked image (tracking image) with a marker that enables visual identification of the individuals of the mice X1 and X2 identified by the individual identification unit 54 on the unit image. The marker is, for example, an individual ID (character) or a tracking point for each individual. The object is a contour line, a frame, a mask image, etc. colored in a different color for each individual.

[0038] The behavior determination unit 56 determines the behaviors of one or more mice X1 and X2 identified by the individual identification unit 54. Here, the behaviors include, in addition to the behaviors of the respective mice X1 and X2, the interaction behaviors between the mice X1 and X2 existing in plurality within the unit image. That is, when the transition of the position of the part in the data frame satisfies any one of one or more preset conditions, the behavior determination unit 56 determines the behavior corresponding to the satisfied condition.

[0039] Specifically, the behavior determination unit 56 determines, for each animal identified (classified) by the individual identification unit 54, whether or not the transition of the coordinates of the part conforms to a preset condition based on the selected mathematical model among the plurality of types of mathematical models in the model DB42 and the material data in the material DB43, and assigns the behavior label of the animal corresponding to the satisfied condition (if it is a scratching behavior, then "01", if it is in a sleeping state, then "02", if it is a grooming behavior, then "03", etc.) to the time information (timestamp) of the image frame including the part or the data frame.

[0040] Thus, for example, in an environment where two mice X1 and X2 are housed and active in cage C, if the frequency of contact between the nose (part) of one mouse X1 and the nose (part) of the other mouse X2 exceeds a predetermined number of times within a certain period, it can be deduced that the animals are engaging in grooming behavior in an attempt to build a friendly relationship, or in the case of male and female, they are attempting to enter into reproductive behavior, etc.

[0041] The behavior determination unit 56 includes a behavior detection unit 81 and a behavior prediction unit 82. The behavior detection unit 81 detects behaviors related to the positional relationship when the positional relationship between a part of an animal and another specific part (drinking water area, feeding area, part of another animal) satisfies the specified conditions. Specifically, the behavior detection unit 81 refers to the model DB 42 and the material DB 43, and checks whether the positional relationship between the parts of mice X1 and X2 and the positional relationship between mice X1 and X2 and other living-related members (drinking water area, feeding area, etc.) in cage C at any point in the data frame satisfy the conditions of the material DB 43, and detects the behaviors corresponding to the satisfied conditions. It detects at least one of the sociality of each of the mice X1 and X2, the interaction between the mice X1 and X2 existing in the same activity range, and the relationship between the mice X1 and X2.

[0042] Based on the involvement between mice X1 and X2 detected by the behavior detection unit 81, the behavior prediction unit 82 predicts how the mice X1 and X2 will live in the future.

[0043] Next, with reference to FIG. 4, the image processing executed by the information processing apparatus will be described. FIG. 4 is a flowchart for explaining an example of the flow of image processing executed by the information processing apparatus having the functional configuration of FIG. 3. In the information processing system of the first embodiment, one or more mice X1 and X2 that are active in cage C are imaged by camera 1, and the video is input to the image processing apparatus 2. As a result, the image processing apparatus 2 executes the following image processing to assist in identifying each individual of the one or more mice X1 and X2 and determining the personality, sociality, etc. of each of the mice X1 and X2.

[0044] In step S11, the image acquisition unit 51 acquires an image (such as a video) composed of a plurality of unit images arranged in the time direction, which is obtained as a result of imaging the state in which one or more animals are active within a certain activity range.

[0045] In step S12, the part extraction unit 52 extracts parts (such as eyes, nose, ears, feet, parts of bones and joints, the center of gravity of the body, etc.) of each part of the body of one or more animals for each of the plurality of unit images included in the acquired video.

[0046] In step S13, the individual identification unit 54 analyzes the plurality of parts extracted from each of the plurality of unit images by the part extraction unit 52 for each time series, and based on the analysis result, identifies each individual of one or more animals in each of the plurality of unit images.

[0047] To identify an individual means, for example, as shown in FIGS. 1 and 3, among the mice X1 and X2 housed in the cage C, an individual ID such as "0" is assigned to the parts of the mouse X1 to identify it as the individual "0", and an individual ID such as "1" is assigned to the parts of the mouse X2 to identify it as the individual "1".

[0048] In step S14, the behavior determination unit 56 determines the behaviors of the one or more mice X1 and X2 identified by the individual identification unit 54.

[0049] In the behavior determination unit 56, the behavior detection unit 81 detects at least one of the sociality of each of the mice X1 and X2, the interaction between the mice X1 and X2 existing in the same activity range, and the relationship between the mice X1 and X2 based on the positional relationship between the parts of the mice X1 and X2 at an arbitrary point in time.

[0050] According to the operation of the image processing apparatus 2 in this way, from the video captured of the activities of two mice X1 and X2 in the cage C, parts of the bodies of the respective mice X1 and X2 are extracted, and the extracted parts are analyzed for each time series to identify the respective individuals of the mice X1 and X2, and a tracking image with markers such as individual IDs and tracking points and objects for individual identification attached to each of the identified individuals is displayed, so that the respective individuals of the mice X1 and X2 that are present and active in the same image can be distinguished. Also, as the behaviors of the mice X1 and X2, for example, the sociality of the mice X1 and X2, the interaction between the mice X1 and X2 present in the same activity range, the relationship between the mice X1 and X2, etc. are detected, so that the dominant-subordinate relationship (dominance, subordination, etc.), habits, ecology, etc. of the mice X1 and X2 can be considered.

[0051] Subsequently, with reference to FIGS. 5 to 11, the image processing procedure of the image processing apparatus 2 in this information processing system will be described. FIG. 5 is a diagram showing an example of a frame image acquired from a video. FIG. 6 is a diagram showing an example of a frame image acquired from the video after the frame image of FIG. 5. FIG. 7 is a diagram showing an example of a frame image acquired from the video after the frame image of FIG. 6. FIG. 8 is a diagram showing a state where individuals are identified from the frame image of FIG. 5. FIG. 9 is a diagram showing a state where parts are detected from the frame image of FIG. 8. FIG. 10 is a diagram showing a state where the parts of FIG. 9 are extracted. FIG. 11 is a diagram showing a state where the parts extracted from the frame images of FIGS. 5 to 8 are tracked respectively. FIG. 12 is a diagram showing a tracking image with markers attached to the frame image at the time t3 in FIG. 8.

[0052] In this information processing system, the inside of the cage C is imaged by the camera 1, and the captured video is transmitted to the image processing apparatus 2. In the image processing apparatus 2, by the image acquisition unit 51, from the video input from the camera 1, as shown in FIGS. 5 to 7, frame images G1, G2, and G3 are acquired following the time series in the order of times t1, t2, and t3.

[0053] Subsequently, in the part extraction unit 52, the unit images (pixels) included in the frame image G1 at time t1, for example, are binarized by the individual recognition unit 71, and the background (the cage part) and the body parts of the white mice X1 and X2 are partitioned, and regions are recognized for each individual. In this example, as shown in FIG. 8, the region 81-1 of the body part of the mouse X1 and the region 91-1 of the body part of the mouse X2 are recognized.

[0054] Next, as shown in FIG. 9, from each of the regions 81-1 and 91-1, one or more parts 82-1 and 92-1 of each of the regions 81-1 and 91-1 as tracking points for tracking the behavior are detected by the part detection unit 72.

[0055] In the example of FIG. 9, the center of gravity of the region 81-1 is calculated by the part detection unit 72, and the two-dimensional coordinates (x1, y1) indicating the part 82-1 of the center of gravity of the region 81-1 on the frame image G1 are detected as tracking points. The part 82-1 is assigned "0" as the individual ID.

[0056] Similarly for the region 91-1, the center of gravity is calculated, and the two-dimensional coordinates (x2, y2) indicating the part 92-1 of the center of gravity of the region 91-1 on the frame image G1 are detected as tracking points. The part 92-1 is assigned "1" as the individual ID. In this way, one or more parts of the animal are detected, and the movement of the body is digitized from the detected parts as the detection result.

[0057] Note that the method of obtaining the center of gravity of the region and using its position as a tracking point as in this embodiment is an example. In addition to this, one or more of the left and right eyes, nose, left and right front feet, left and right hind feet, joints of each foot, the outline and center of the ears, the backbone, the center and the hairline and tip of the tail, etc. of the animal may be detected and used as tracking points.

[0058] By detecting one or more parts of an animal in this way, tracking the parts over time, and comparing them with the learning model of the model DB42, it is possible to identify an animal having a skeleton that moves in such a way, that is, to identify an individual such as what kind of animal the animal is.

[0059] The body parts (tracking points) detected in this way are extracted, and as shown in FIG. 10, tracking data 101 including the extracted parts 82-1, 92-1 and the individual ID is generated.

[0060] The process of generating the above tracking data 101 is also performed for the frame images G2 and G3, and by analyzing each time series, it is possible to convert into data the positional relationship of each part of each individual of each mouse X1 and X2 and the frequency of satisfying certain conditions at any point in time.

[0061] For example, as shown in FIG. 11, by overlapping the tracking data 101 having the parts generated corresponding to the frame images G1 to G3, an action trajectory accompanying the action of the animal can be obtained. That is, the action trajectory of the mouse X1 is obtained as vector data 111-1 and 111-2 indicating the trajectories of the parts 82-1, 82-2, and 82-3 moving. Also, the action trajectory of the mouse X2 is obtained as vector data 112-1 and 112-2 indicating the trajectories of the parts 92-1, 92-2, and 92-3 moving.

[0062] Then, the marker image generation unit 55 generates an image in which a marker indicating a part of the animal and an individual ID or the like are superimposed on a unit image, and as shown in FIG. 12, a tracking image G3A is displayed. In this tracking image G3A, for example, a marker (round mark) indicating the part 82-3 of the mouse X1, an individual ID "0", and a trajectory line 120 are added to the frame image G3 (see FIG. 7) at the time point of time t3. Also, in the tracking image G3A, a marker (triangle mark) indicating the part 92-3 of the mouse X2, an individual ID "1", and a trajectory line 121 are displayed.

[0063] By playing back the tracking images while holding the trajectory lines 120 and 121 from the frame images G1 and G2 at times t1 and t2 for a predetermined time (for example, about 0.5 seconds), it is possible to play back a video in which the mice X1 and X2 move while leaving the trajectory lines 120 and 121 displayed.

[0064] Here, with reference to FIGS. 13 to 15, some examples of determining the behavior of each individual mouse from the positional relationship of one or more parts of the mouse and detecting sociality, interaction, and relationship will be described. FIG. 13 is a diagram showing a detection example of the water-drinking behavior of a mouse. FIG. 14 is a diagram showing a detection example of the food-eating behavior of a mouse. FIG. 15 is a diagram showing a detection example of the interference behavior between mice.

[0065] As shown in FIG. 13, in the image processing apparatus 2, from the video captured by the camera 1 of the cage containing the mouse, specific parts of the mouse (in this example, the nose and the left and right eye parts) are extracted as tracking points, and as a result of tracking the positions (coordinates x, y) of the tracking points over time, coordinates indicating the positions of the nose and the left and right eye parts of the mouse are obtained for each individual mouse for each frame image.

[0066] In the image processing apparatus 2, the behavior determination unit 56 determines the behavior of the animal based on whether the time-series change in the position of a predetermined part of the animal matches a predetermined condition. Specifically, in a plurality of frames, for example, frames No. 150 to 156 in FIG. 13 (the bracketed part), when the distance between the position (coordinates) of the mouse's nose and the position (coordinates) of the water-drinking opening attached to the cage wall is zero or close to zero and this proximity state continues, the behavior determination unit 56 determines that the mouse is performing a water-drinking behavior.

[0067] Furthermore, in frames No. 150 to 156 in FIG. 14 (the bracketed part), when the distance between the position (coordinates) of the mouse's nose and the position (coordinates) of the food tray placed in the cage is zero or close to zero and this proximity state continues, the behavior determination unit 56 determines that the mouse is performing a food-eating behavior.

[0068] Also, in the frames No. 150 to 156 in FIG. 15 (the part in parentheses), when the distance between the positions (coordinates) of the mouse's nose and the position (coordinates) of the mouse's nose is zero or in a proximity state close to zero and continues, the behavior determination unit 56 determines that it is an interference behavior between the mice.

[0069] As described above, according to the image processing apparatus 2 in the information processing system of the first embodiment, from the video in which the activities of two mice X1 and X2 in the cage C are imaged, a part of each body of the mice X1 and X2 is extracted, and the positions of the extracted parts are analyzed for each time series to identify the individuals of the mice X1 and X2, respectively. Therefore, the individuals of the mice X1 and X2 can be distinguished from the video in which the mice X1 and X2 that are active within a certain behavior range are imaged. Also, regarding the behavior of the mice, for example, the drinking behavior of the mice, the eating behavior of the mice, the interference actions between the mice existing in the same behavior range, etc. are determined, so that the habits, ecology of the mice, and the dominant-subordinate relationship (dominance, subordination, etc.) between the mice can be considered. In the individual identification unit 54, based on the premise that the part positions do not greatly deviate between the front and rear frames, the individuals can be correctly identified from the continuity of the part positions extracted in the previous stage. In the individual identification unit 54, since the individuals are identified based on the outer contour and the position information of the parts, the individuals can be correctly identified without depending on the brightness of the shooting environment and the background.

[0070] Subsequently, a second embodiment will be described with reference to FIG. 16. In describing this second embodiment, the same components as the functional configuration of the first embodiment shown in FIG. 3 are denoted by the same reference numerals and their description is omitted. FIG. 16 is a functional block diagram showing a second embodiment of the information processing system of FIG. 1, that is, a second embodiment of the functional configuration of the image processing apparatus 2 of FIG. 2.

[0071] The image processing apparatus 2 includes an outer contour detection unit 57. The outer contour detection unit 57 uses a selected skeletal estimation model from among a plurality of types of skeletal estimation models in the model DB 42 to detect the outer contour (such as the body contour) of each body of one or more animals for each of a plurality of unit images. The outer contour detection unit 57 includes an individual recognition unit 91 and an outer contour specification unit 92. The individual recognition unit 91 recognizes, in each of a plurality of unit images, a region including a unit image with change (movement) as one acting individual. Specifically, the individual recognition unit 91 binarizes each unit image and recognizes as part of the acting individual those with a color different from the background image. Preferably, the boundary of the region is set as the portion where the color changes as compared with the frame images before and after in the time series. The outer contour specification unit 92 specifies, for each of a plurality of unit images, the region including what has been recognized as one acting individual as the outer contour (body contour, etc.) of each of one or more mice X1, X2. The data frame generation unit 53 generates a data frame (see FIG. 22) indicating the probability that the change in the outer contour accompanying the behavior of the animal is a specific behavior.

[0072] The individual identification unit 54 identifies the individuals of one or more animals in each of a plurality of unit images based on the output obtained by inputting the time series of the outer contours of each of one or more animals detected from each of a plurality of unit images by the outer contour detection unit 57 to the skeleton estimation model of the model DB42. The individual identification unit 54 analyzes the one or more outer contours detected from each of a plurality of unit images by the outer contour detection unit 57 for each time series, and identifies the individuals of one or more animals in each of a plurality of unit images based on the analysis result. Specifically, the individual identification unit 54 analyzes the one or more outer contours detected from each of a plurality of frame images with reference to the model DB42 for each time series, thereby identifying the individuals of mice X1 and X2 respectively, and outputs the information (individual ID and position of the outer contour, etc.) of the identified individuals of mice X1 and X2 to the behavior determination unit 56. That is, the individual identification unit 54 identifies which of the two mice X1 and X2 in the cage the detected outer contour belongs to.

[0073] The learning model of the model DB42 in this second embodiment stores learned training data for identifying a mouse and outputting individual information (such as individual ID and the position of the outline) of the identified individual when a new video or still image is input, as a result of performing machine learning on videos and still images for each individual mouse prepared in advance.

[0074] Specifically, for example, when a video obtained by photographing the activities of each mouse is input to the model DB42, it outputs the individual ID of each of one or more mice included in the video and the position of the outline (body contour) of the mouse having that individual ID in the video. That is, by using the learning model in this second embodiment, it is a learning model using a skeleton estimation method, and the position of the outline of an individual can be detected from the area of each identified individual.

[0075] Based on the individual information input from the individual identification unit 54, the behavior determination unit 56, for example, quantifies the general behavior of each individual at an arbitrary time point and the frequency of satisfying certain conditions, and collates the behavior and conditions of the individual with the data in the material DB43. Specifically, when the behavior determination unit 56 satisfies any one of one or more conditions in which the probability value of a specific behavior in the data frame is preset, it determines the behavior corresponding to the satisfied condition. Here, the one or more conditions are, for example, a condition that after one mouse performs a general behavior of rounding its body, the other mouse performs a general behavior of approaching the body of the mouse that has rounded its body, and the frequency is a certain number of times or more. The behavior determination unit 56 refers to the material DB43 and, when the behavior of the mouse identified by the individual identification unit 54 satisfies the condition, determines that the relationship between the mice is good and that they are paired.

[0076] According to the functional configuration of the image processing apparatus 2 in the information processing system of the second embodiment as described above, the same effects as those of the first embodiment are achieved, and the following effects are obtained. That is, from a video in which the states of two mice X1 and X2 housed and active in the cage C are imaged, the outer contours of the bodies of the respective mice X1 and X2 are detected, the detected outer contours are analyzed for each time series, and the individuals of the respective mice X1 and X2 are identified. Thus, the individuals of the mice X1 and X2 that are present and active in the same image can be distinguished from each other.

[0077] (Second Embodiment) Next, with reference to FIG. 17, the image processing operation executed by the information processing apparatus having the functional configuration of the second embodiment in FIG. 16 will be described. In describing the image processing operation of the second embodiment, the same operations as those in the first embodiment shown in FIG. 4 are given the same step numbers as in FIG. 4 and the description thereof is omitted. FIG. 17 is a flowchart for explaining an example of the flow of image processing executed by the information processing apparatus having the functional configuration of FIG. 16.

[0078] In the case of the second embodiment, in step S21, when a video is acquired from camera 1, in step S22, the outer contour detection unit 57 detects the outer contour (such as the contour of the body) of each body of one or more animals for each unit image of a plurality of frame images included in the acquired video.

[0079] In step S23, the individual identification unit 54 analyzes the outer contours extracted from each of the plurality of unit images by the outer contour detection unit 57 for each time series, and based on the analysis result, identifies the individuals of one or more animals in each of the plurality of unit images. The processing after step S24 is the same as that in the first embodiment (FIG. 4).

[0080] According to the operation of the image processing apparatus 2 having the functional configuration of the second embodiment, for each unit image of a plurality of frame images included in the video acquired from camera 1, the outer contours of the bodies of the mice X1 and X2 are detected, and by analyzing how the outer contours change following the actions of the mice X1 and X2 over time, the individuals of the mice X1 and X2 are identified. Thus, the individuals of the mice X1 and X2 shown in the same image can be distinguished from each other. In addition, by monitoring how the outlines of the identified mice X1 and X2 change over time series and satisfying predetermined conditions, specific behaviors of the mice X1 and X2 are detected, so that not only existing ecologies but also new ecologies can be discovered.

[0081] Subsequently, with reference to FIGS. 18 to 22, the image processing procedure of the information processing apparatus in this information processing system will be described. FIG. 18 is a diagram showing an example of a frame image obtained from a video. FIG. 19 is a diagram showing an example of a frame image obtained from the video after the frame image of FIG. 18. FIG. 20 is a diagram showing a tracking image with an object indicating an outline and an individual marker attached to the frame image at time t4 in FIG. 18. FIG. 21 is a diagram showing a tracking image with an object indicating an outline and an individual marker attached to the frame image at time t5 in FIG. 19. FIG. 22 is a diagram showing a detection example of a scratching behavior. In addition, when explaining FIGS. 18 to 21, among the two mice shown in the frame image, for one mouse, the mouse included in the frame image G4 at time t4 is described as mouse X1-1, and the mouse included in the frame image G5 at time t5 is described as mouse X1-2. The same applies to the other mice X2-1 and X2-2.

[0082] In this information processing system, the inside of the cage C is imaged by the camera 1, and the captured video is transmitted to the image processing apparatus 2. In the image processing apparatus 2, the image acquisition unit 51 sequentially acquires the frame image G4 at time t4 shown in FIG. 18 and the frame image G5 at time t5 shown in FIG. 19 from the video input from the camera 1 in time series.

[0083] Subsequently, in the outline detection unit 57, the unit image included in the frame image G4 at time t4, for example, is binarized by the individual recognition unit 91, so that the background (the cage part) and the white mouse body part become regions of different colors, and regions are recognized for each individual. As a result, the regions of the body parts of one of the two mice are individually recognized.

[0084] FIG. 20 shows a display example of the tracked image G4A that has been individually identified. In this display example of the tracked image G4A, a blue object 121-1 is displayed on the outline (body contour) of mouse X1-1, and a red object 122-1 is displayed on the outline (body contour) of mouse X2-1. In addition, on the same tracked image G4A, markers such as an individual ID “mouse0.996” for identifying the individual of mouse X1-1 and a frame 131-1 surrounding the individual are displayed. Furthermore, on the same tracked image G4A, markers such as an individual ID “mouse0.998” for identifying the individual of mouse X2-1 and a frame 132-1 surrounding the individual are displayed.

[0085] Similarly, in the frame image G5 at time t5, the unit images (pixels) are binarized, the background (cage part) and the white mouse body part are partitioned, and regions are recognized for each individual.

[0086] In the example of the individually identified tracked image G5A in FIG. 21, a blue object 121-2 is displayed on the outline (body contour) of mouse X1-2, and a red object 122-2 is displayed on the outline (body contour) of mouse X2-2. In addition, on the same tracked image G5A, markers such as an individual ID “mouse0.996” for identifying the individual of mouse X1-2 and a frame 131-2 surrounding the individual are displayed. Furthermore, on the same tracked image G5A, markers such as an individual ID “mouse0.998” for identifying the individual of mouse X2-2 and a frame 132-2 surrounding the individual are displayed.

[0087] In this way, the outlines of the body parts of the two mice are detected, and from the situation where the outlines change over time, after identifying the individuals of the mice, when each individual satisfies a predetermined condition at an arbitrary time, a specific behavior of the mouse is determined. Therefore, the reasons for the behavior of each mouse can be known.

[0088] Here, with reference to FIG. 22, an example of determining the behavior of each individual mouse from the positional relationship of the outer contour of the mouse and detecting sociality, interaction, and relationship will be described. FIG. 22 is an example of detecting scratching behavior. Frame images are acquired in order from Frame No. 1 of FIG. 22. After identifying the mouse, the behavior determination unit 56 calculates the probability of whether the mouse is performing a specific behavior for each frame image, and frames the probability obtained as a calculation result for each frame image into a data frame. Then, based on the probability of the data frame, the behavior of the mouse is labeled (classified), and the identification information is attached to the data frame and stored in the image DB 41.

[0089] For example, in the data frames of Frame No. 4502 to 4507 in FIG. 22, since the calculated probability exceeds a certain threshold value, the behavior determination unit 56 determines that the mouse has performed a specific behavior, and attaches identification information indicating the specific behavior (for example, predict “1”, etc.) to the data frames of Frame No. 4502 to 4507. When the time-series change of the outer contour of the mouse (the behavior of the mouse) matches the conditions set in the material DB 43 in advance, a specific behavior of the mouse derived from the conditions, for example, the behavior of the mouse scratching the floor of the cage, that is, the scratching behavior, is determined.

[0090] According to the image processing apparatus 2 of the second embodiment, the probability of whether the identified mouse is performing a specific behavior is calculated for each frame image, and based on the probability obtained as a calculation result, the data frame at the time when the mouse performs a specific behavior is labeled. Therefore, not only can the established behavior of the mouse be detected, but also unexpected behaviors can be detected. As a result, while distinguishing each of the plurality of mice housed in the cage and observing them, it is possible to discover general behaviors and new behaviors performed by each mouse.

[0091] (Third Embodiment) Next, the third embodiment will be described with reference to FIGS. 23 to 30. The above-described first and second embodiments only constructed models for specific animal species and behaviors (such as white mice, scratching behaviors, etc.) and shooting environments, output them as RAW data (position information of each part, etc.), and only constructed a condition determination program. However, this third embodiment expands the application to multiple animal species, behaviors, and shooting environments, enhancing practicality. In the third embodiment, annotation for RAW data acquisition (creation of teacher data for determining the position of the acquisition site), construction of determination conditions (mathematical models) for each behavior using RAW data, programming and modularization for application, and expansion of programs for detecting the movement, feeding, and drinking of white mice were performed.

[0092] Also, the above-described first and second embodiments realized the minimum necessary tasks in their own hardware and software environments. However, in the third embodiment, to enhance versatility and enable stable use by general users with low IT literacy, a user interface environment (hereinafter referred to as the "UI environment") that is easy to operate with consistency in the business process was constructed. Also, assuming that the execution environment of the system operates stably without depending on the conditions of the user (client) side, and that the specifications and costs are optimized according to the frequency and load of the execution tasks, the system was constructed as a cloud service, that is, a server-client system.

[0093] First, referring to FIG. 23, the outline of the information processing system of the third embodiment will be described. FIG. 23 shows the third embodiment of the information processing system, and is a diagram showing the outline of a business model obtained by commercially expanding the information processing systems of the first and second embodiments.

[0094] As shown in FIG. 23, the information processing system of the third embodiment is configured such that a device on the client 300 side of the client Y and a server 200 of the contractor who undertakes the request communicate with each other. The server 200 receives a request from the client Y, analyzes the video uploaded from the client Y, and transmits the analysis result to the client Y.

[0095] The devices on the client 300 side include a camera 310 and a cage 311 that constitute the animal shooting environment, and a client computer 320 (hereinafter referred to as "PC 320") that collects the video of the analysis target imaged by the camera 310. The cage 311 is a container for allowing an animal to move within a certain range. The camera 310 images the animal moving within the cage 311. This animal shooting environment can be appropriately changed according to the request content.

[0096] The PC 320 captures the video (image) data imaged by the camera 310 and stores it locally (such as in an internal storage). The PC 320 uploads the video data 321 stored locally to the server 200 and requests an analysis of the actions of the animals included in the video. The PC 320 obtains information on the analysis results (CSV file 330) from the server 200 for the request and video data (processed video 331) with markers M attached to the positions of the animal's eyes after processing. The PC 320 creates materials (graph 322) for analyzing the data in the CSV file 330 or uses the processed video 331 as part of the analysis materials.

[0097] The server 200 includes an image analysis unit 450, a web service unit 451, a storage 410, and a data warehouse 411 (hereinafter referred to as "DWH 411"). The image analysis unit 450 includes the functional configuration shown in FIG. 3 of the first embodiment and the functional configuration shown in FIG. 16 of the second embodiment, and performs image analysis processing.

[0098] The web service unit 451 has functions of authenticating the login information input from the PC 320, searching for the analysis target, outputting the search results, adding analysis data, and displaying the analysis results. Specific functional configurations will be described later. In addition, the web service unit 451 has functions of performing encrypted communication, access source IP communication, dedicated line communication, etc., so as to implement security measures. That is, the web service unit 451 realizes the interface with the PC 320.

[0099] The storage 410 stores the video data requested for analysis and the video data being analyzed through the web service unit 451. The DWH 411 stores various related data, and systematically archives the results processed in cooperation with the data in the storage 410. This makes the data reusable and can reduce the number of animal experiments.

[0100] Here, the data processing procedure (flow) of the information processing system of this third embodiment will be described. In step S101, the state of the animal's activity in the cage 311 is imaged by video, and the video data 321 is stored locally in the PC 320. In step S102, when requesting the contractor to analyze the video, the video data 321 stored in the PC 320 is uploaded to the server 200.

[0101] In the server 200, the web service unit 451 stores the video data 321 uploaded from the PC 320 in the storage 410. In steps S101 and S102, the captured video data 321 is temporarily stored in the PC 320 and then uploaded to the server 200. However, in the case of a video captured continuously for 24 hours, the video data 321 captured by the camera 310 may be directly uploaded to the server 200 as in step S103.

[0102] In step S104, in the server 200, the image analysis unit 450 reads out the video data stored in the storage 410, executes the analysis of the requested video data 321, stores the processed video 331 generated by processing the video during the analysis and the CSV file 330 of the analysis result in the storage 410, and outputs an analysis completion notification to the PC 320. Upon receiving the analysis completion notice, on the PC 320, when the search screen is displayed and the video data to be analyzed is specified, in step S105, the web service unit 451 searches for the specified video data and downloads the processed video 331 and its analysis result CSV file 330 to the PC 320.

[0103] In step S106, the PC 320 creates a graph 322 for analysis using the processed video 331 and its analysis result CSV file 330 downloaded from the server 200, and attaches the processed video 331 as supplementary material for the graph 322.

[0104] Note that in the above steps S105 and S106, the analysis result CSV file 330 (numerical data) was downloaded to the PC 320, and the graph 322 was created from the CSV file 330. However, in addition, for example, as in step S107, depending on the request content, on the server 200, the graph 322 can be created from the CSV file 330 and the graph 322 can be downloaded to the PC 320 as the analysis result.

[0105] Next, referring to FIG. 24, the functional configuration of the information processing system according to the third embodiment of FIG. 23 will be described. FIG. 24 is a functional block diagram showing the functional configuration of the information processing system according to the third embodiment of FIG. 23. In describing the functional configuration of the information processing system according to the third embodiment, the same components as those in the functional configuration of the first embodiment shown in FIG. 3 and the functional configuration of the second embodiment shown in FIG. 16 are denoted by the same reference numerals and their description is omitted.

[0106] As shown in FIG. 24, an authentication DB 44 is stored in an area of the storage unit 18 of the server 200. The authentication DB 44 stores authentication information for the requester Y to log in to the server 200. The authentication information is a login ID and a password, etc. as identification information for identifying the requester Y. In addition, the model DB 42 stores in advance a plurality of types of learning models generated or updated by machine learning or the like. The learning models are, for example, mathematical models such as a skeleton estimation model and an individual identification model. The plurality of types of learning models are provided, for example, for each type of animal to be analyzed, for each imaging direction of the image, and for each color of the animal.

[0107] When the processes corresponding to steps S101 to S107 in FIG. 23 are executed, in the CPU 11 of the server 200, the image analysis unit 450, the web service unit 451, and the process unit 452 function. The image analysis unit 450 performs image analysis processing on the designated image data among the video data 321 acquired according to the request from the requester Y using the selected model. The image analysis unit 450 reads out the image to be analyzed designated by the designation unit 471 from the image DB 41, and uses the learning model (such as a skeleton estimation model, an individual identification model, a mathematical model, etc.) selected from among the plurality of types of learning models by the model selection unit 472 and extracted from the model DB 42 to execute the analysis processing of the image. The image processing executed by the image analysis unit 450 is the processing such as individual identification, outline identification, and behavior analysis of animals in the video shown in the first embodiment and the second embodiment. The web service unit 451 performs login authentication when the requester Y logs in to the server 200, searches for the results of video analysis requested by the requester Y, outputs the search results, adds analysis data, controls the display of the analysis results, and the like.

[0108] The web service unit 451 includes an authentication unit 461, a search unit 462, a search result output unit 463, an analysis data addition unit 464, an analysis result display control unit 465, and the like.

[0109] The authentication unit 461 performs user authentication by comparing the input login information with the authentication information in the authentication DB. When the information matches as a result of comparing the mutual information, the authentication unit 461 permits the requester Y to log in to the server 200.

[0110] The search unit 462 displays a search screen 251 (see FIG. 25) on the PC 320 of the authenticated requester Y, searches for analysis results from the image DB 41 according to a search request from the search screen 251, and passes the search results to the search result output unit 463. On the search screen 251, a search result list, addition of analysis data, display of analysis results, etc. can be performed.

[0111] The search result output unit 463 outputs the search results retrieved by the search unit 462 to the PC 320 and displays them as a list on the search screen 251 of the PC 320.

[0112] When an analysis data addition operation is performed on the search screen 251, the analysis data addition unit 464 displays an analysis data addition screen 261 (see FIG. 26) on the PC 320. Through the analysis data addition operation from this analysis data addition screen 261, the video data to be analyzed is uploaded to the image processing apparatus 2. In the image processing apparatus 2, the uploaded video data to be analyzed is additionally registered in the image DB 41. After the upload, on the analysis data addition screen 261, the analysis conditions for the new file added to the image DB 41 can be specified.

[0113] The analysis data addition unit 464 includes a specification unit 471 and a model selection unit 472. The specification unit 471 specifies analysis attributes for the image to be analyzed (type of the object to be analyzed (e.g., mouse, rat, etc.), imaging direction of the object to be analyzed (e.g., upward, obliquely upward, horizontal, etc.), color of the object to be analyzed (e.g., white, black, etc.)).

[0114] Specifically, the specification unit 471 displays the analysis data addition screen 261 of FIG. 26. On the analysis data addition screen 261, a file column and a setting column are provided. In the file column, an icon of the file of the image to be analyzed and a bar graph indicating the progress of the analysis (progress status) are provided. In the setting column, there are radio buttons (designation buttons) for the requester Y to specify analysis attributes such as the species of the animal included in the image to be analyzed (e.g., mouse, rat, etc.), the imaging direction of the animal to be analyzed (e.g., top, diagonally upward, horizontal, etc.), the color of the animal to be analyzed (e.g., white, black, etc.).

[0115] When the requester Y uploads the image to be analyzed to the image processing device 2, an icon of the image file to be added, which has been uploaded, is displayed in the file column of the analysis data addition screen 261.

[0116] Then, by the requester Y performing image settings (such as the species of the animal included in the image to be analyzed, the imaging direction of the animal to be analyzed, the color of the animal to be analyzed, etc.) using the radio buttons provided in the corresponding setting column below the icon of the image file, the accuracy of image analysis during image analysis can be improved, and the species and individual identification of the animals included in the image, behavior analysis, etc. can be accurately performed.

[0117] Based on the analysis attributes of the image specified by the specifying unit 471, the model selection unit 472 selects a model to be applied to the part extraction unit 52 in FIG. 3 or the outline detection unit 57 in FIG. 16 from among a plurality of types of skeleton estimation models stored in the model DB, and also selects a model to be applied to the individual identification unit 54 from among a plurality of types of individual identification models stored in the model DB. The analysis result display control unit 465 displays the CSV file 330 of the analysis result, the processed video 331, etc. downloaded from the server 200.

[0118] The process unit 452 manages image data and unprocessed data for analysis. The process unit 452 includes an upload data management unit 491 and an unprocessed data management unit 492. The upload data management unit 491 moves the image data uploaded from the PC 320 to the image DB 41 and updates the management file. In addition, the upload data management unit 491 has functions such as a function to be started every minute by cron and a function to prevent duplicate startups.

[0119] The unprocessed data management unit 492 checks the unprocessed data in the management file. The unprocessed data management unit 492 copies the unprocessed data. The unprocessed data management unit 492 performs AI processing in the processing directory. The unprocessed data management unit 492 stores the result file in the corresponding directory. The unprocessed data management unit 492 creates the CSV file 330 of the analysis result and stores it in the corresponding directory. In addition, similar to the upload data management unit 491, the unprocessed data management unit 492 has functions such as a function to start every minute by cron, a function to prevent duplicate startup, and a function to update the management file.

[0120] When the processes corresponding to steps S101 to S107 in FIG. 23 are executed, in the CPU 141 of the PC 320, the login management unit 421, the screen control unit 422, the analysis request unit 423, and the analysis result display control unit 424 function. The login management unit 421 displays a login screen on the PC 320, and transmits the login information input by the requester Y on the login screen to the server 200 to request login authentication. The screen control unit 422 displays a search screen on the PC 320, transmits the search keyword input by the requester Y on the search screen to the server 200 to request a search, and displays the search result for the request on the search screen. A search result list is displayed on the search screen. In addition, on the search screen, analysis data can be added, the analysis result can be displayed, etc. The analysis request unit 423 displays an analysis data addition screen on the PC 320, and transmits the file and analysis conditions specified by the requester Y on the analysis data addition screen by the requester Y to the server 200 to request an analysis. The analysis result display control unit 424 displays an analysis result display screen on the PC 320, and displays the CSV file 330 of the analysis result and the processed video 331 downloaded from the server 200 on the analysis result display screen.

[0121] Hereinafter, the operation of the information processing system according to the third embodiment will be described with reference to FIGS. 25 to 27. FIG. 25 is a diagram showing a search screen displayed on a PC. FIG. 26 is a diagram showing an analysis data addition screen that is pop-up displayed above the search screen. FIG. 27 is a diagram showing a video confirmation screen that is pop-up displayed above the search screen.

[0122] The search screen shown in FIG. 25 is displayed on the PC 320 (see FIG. 24). The search screen 251 is provided with an input field for inputting search conditions such as keywords and periods, an analysis data list, an analysis data addition button, a video confirmation button, a CSV output button, and the like. The analysis data list is a list of analysis data in each item such as the file name, registrant, date and time, and analysis of the analysis data. The analysis item indicates the current analysis state of the analysis data, and for example, "analyzed", "executing", etc. are shown. On the left side of the analysis data list, selection buttons for selecting each of the analysis data are provided, and by performing button operations on the selected analysis data, addition of analysis data, confirmation of video, CSV output, etc. are possible.

[0123] On this search screen 251, when a large number of analysis data are registered in the server 200, the analysis data are narrowed down by the search conditions, and after selecting (designating) the analysis data with the selection button from those displayed in the analysis data list, by pressing any one of the analysis data addition button, video confirmation button, and CSV output button, processing corresponding to the button operation (addition of analysis data, confirmation of video, CSV output, etc.) can be performed.

[0124] (Addition of analysis data) On the search screen 251 of FIG. 25, after selecting (designating) the desired analysis data with the selection button, for example, when the analysis data addition button is pressed, the analysis data addition screen 261 shown in FIG. 26 is pop-up displayed above the search screen 251.

[0125] The analysis data addition screen 261 is provided with a file of the analysis data to be added, a setting field for analysis attributes for setting (designating) what kind of videos are included in each file, a return button, a cancel button, a registration button, and the like. In the setting field for analysis attributes, buttons for selecting, for example, the type of animal, the imaging direction of the animal, the color of the animal, etc. are provided, and settings (specifications) can be made using each button. On this analysis data addition screen 261, for the video data uploaded as new analysis data, by performing file selection and settings for analysis targets, subsequent analysis of the analysis data can be automatically executed. At this time, based on the type of animal, the imaging direction of the animal, the color of the animal, etc. set as analysis attributes on the analysis data addition screen 261, among the multiple types of learning models (skeleton estimation model, individual identification model, and mathematical model) stored in the model DB 42, the learning model to be used for the animal to be analyzed is selected.

[0126] (Video confirmation) On the search screen 251 in FIG. 25, after selecting (specifying) desired analysis data with the selection button and then pressing, for example, the video confirmation button, the video confirmation screen 271 shown in FIG. 27 is popped up and displayed above the search screen 251. The video confirmation screen 271 is provided with a playback area for playing the analysis data to be added, a back button, a download button, etc. In the playback area, an image (still image) where the selected analysis data (video data) has stopped and a playback button (triangle icon) are provided. By clicking the playback button (triangle icon), the still image starts moving and is played as a video. On this video confirmation screen 271, for the video data uploaded as new analysis data, when it is difficult to determine whether it is data to be analyzed or a processed video that has been analyzed only by the file name, the video can be played and confirmed. Also, by pressing the download button, the analysis data displayed in the playback area can be downloaded. This download function is effective when the processed video 331 is to be used on the PC 320 side.

[0127] (Advantages of utilizing this information processing system) Referring to FIG. 28, the advantages of utilizing this information processing system will be described. FIG. 28 is a diagram showing an example of a report created from a video. In this information processing system, it is possible to output quantitative data on the opening and closing of a mouse's eyes that cannot be visually confirmed. As a result, it becomes possible to confirm details with unprecedented accuracy and details that could not be grasped before. For example, as shown in FIG. 28, when video data 281 of a mouse is captured from the PC 320 of requester Y and uploaded to the server 200, and there is a request to analyze the state of the mouse's eyes, the part detection unit 72 sets a marker M at the position of the mouse's eyes. By identifying that the area of the eyes widens when the mouse opens its eyes and narrows when the mouse closes its eyes, that is, by identifying the opening and closing of the eyes, a graph 283 showing the change in the area of the eyes over time, for example, can be created as a report 282 and provided to requester Y. In report 282, it is possible to output data such as the number of times of eye closure per certain time, the duration of eye closure (how many times the eyes are closed in one minute), and other data according to the request content. That is, in this information processing system, it is possible to quantitatively capture changes in extremely small parts. It can automatically process long videos. The throughput can be improved.

[0128] (Robust Security) Referring to FIG. 29, the security of this information processing system will be described. FIG. 29 is a diagram showing a configuration example of a closed network connection that forms robust security. This information processing system can be configured by selecting a security level according to the policies and costs on the user side. Note that encryption of communication is essential and is based on https. Examples of system configurations according to the security level include, for example, "plum", "bamboo", and "pine". "Plum" allows access only through the GIP of the in-house proxy. "Bamboo" allows access via a reverse proxy using the client's certificate. "Pine" makes a closed network connection with the in-house environment and DX (direct connection). The configuration example of the closed network connection of "pine" is shown in FIG. 29.

[0129] (Deepening of this information processing system) The deepening of this information processing system will be described with reference to FIG. 30. FIG. 30 is a diagram showing the dashboard screen of this information processing system. As shown in FIG. 30, this information processing system has a function of displaying a dashboard screen 500 that is easy to manage information. The dashboard screen 500 has features such as being able to be expanded according to the animal species and experimental environment, being able to output in a format other than the experimental report, and being able to discover and notify feature points that are not noticed by humans by artificial intelligence.

[0130] Specifically, on the dashboard screen 500, as various materials necessary for conducting experiments, for example, an application form for approval of the installation of a feed storage facility, an animal experiment plan, an animal experiment result report, self-inspection, and evaluation items can be created. Regarding the customization according to the business, it is possible to expand the applicable animals and experimental contents. It can support output in a format other than the experimental report.

[0131] Also, on the dashboard screen 500, it is possible to expand the "discovery" by deepening machine learning. For example, it is possible to make "discoveries" within the human cognitive range and "discoveries" beyond the human cognitive range. Regarding machine learning and function deepening, it is possible to expand the applicable animals and experimental contents. It is possible to discover and notify feature points that are not noticed by humans by artificial intelligence.

[0132] To sum up, in the information processing system of the third embodiment, when specifying analysis attributes for an image to be analyzed (type of analysis target (e.g., mouse, rat, etc.), imaging direction of the analysis target (e.g., upward, diagonally upward, horizontal, etc.), color of the analysis target (e.g., white, black, etc.)), the model selection unit 472 selects a skeleton estimation model to be applied to the part extraction unit 52 in FIG. 3 and the outer contour detection unit 57 in FIG. 16 from among a plurality of types of skeleton estimation models in the model DB 42 based on the analysis attributes of the specified image of the analysis target, and also selects an individual identification model to be applied to the individual identification unit 54 in FIGS. 3 and 16 from among a plurality of types of individual identification models. According to the analysis instruction of the image, the selected skeleton estimation model and individual identification model are used to analyze the image. Therefore, the type of the animal can be correctly identified using a model suitable for the analysis target, and analysis results such as the behavior of the animal can be obtained. Also, while applying to multiple animal species and experimental contents, by providing functions such as management and download of analysis data in the server 200, the analysis data can be reused, and the number of animal individuals actually used in the experiment can be significantly reduced. In addition, the following effects can be achieved. For example, since labeling (tagging) of the analysis data is automatically performed based on the specified analysis attributes and the like, when reusing the analysis results, it can be searched from the tags, making it easier to search the analysis data. When using only a single model for multiple analysis attributes and the like, it is necessary to retrain to update the existing model and reexamine its accuracy again. However, by holding multiple models for multiple analysis attributes and the like, when a new attribute appears, a new model will be created, so that model learning can be performed compactly.

[0133] The above-described series of processes can be executed by hardware or by software. In other words, the functional configurations in FIGS. 3, 16, and 24 are merely illustrative and are not particularly limited. That is, it suffices that the information processing system is equipped with a function capable of executing the above-described series of processes as a whole. There is no particular limitation to the function blocks and databases used to realize this function, and it is not limited to the examples in FIGS. 3, 16, and 24. Also, the locations of the function blocks and databases are not particularly limited to those in FIGS. 3, 16, and 24, and may be arbitrary. The function blocks and databases of the image processing apparatus 2 and the server 200 may be transferred to the camera 1, the PC 320, or the like. Furthermore, the image processing apparatus 2 and the camera 1 or the PC 320 may be the same hardware.

[0134] For example, when a series of processes are executed by software, the programs constituting the software are installed in a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. Also, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or personal computer in addition to a server.

[0135] For example, a recording medium containing such a program is not only constituted by a removable medium (not shown) distributed separately from the apparatus main body to provide the program to the user, but also constituted by a recording medium or the like provided to the user in a state pre-installed in the apparatus main body.

[0136] Note that in this specification, the steps of describing the program recorded on the recording medium include not only processes performed in time series according to the order, but also processes executed in parallel or individually, even if they are not necessarily processed in time series. Also, in this specification, the term "system" shall mean an overall apparatus composed of a plurality of apparatuses, a plurality of means, and the like.

[0137] In the above-described embodiment, the marker is in the shape of a triangle or a circle, but it is not limited to this example, and other shapes may be used. The same applies to the object indicating the outline of the body. In the above-described embodiment, it has been described that the action determination unit 56 determines whether or not the conditions defining the positional relationship between the part of the animal and other specific parts (drinking water area, feeding area, parts of other animals) are satisfied. However, the conditions defining the positional relationship between the part of the animal and other specific parts are stored in the material DB43 in the storage unit 18, and the action determination unit 56 reads out the above conditions from the material DB43. Additionally, an action database may be provided separately from the material DB43, and the above conditions may be stored in the action database. In the above-described embodiment, a machine learning method is used. However, other rules may be set, for example, the part farthest from the center of gravity is defined as the nose. In the above-described embodiment, the center of gravity is detected (extracted) as a part of the body. However, other parts, for example, specific parts among eyes, nose, ears, feet, bones, and joints (such as eyes, nose, ears, etc.) may be detected (extracted). In the above-described embodiment, a mouse is used as an example of the object for individual identification. However, by expanding the data in the model DB42 and the material DB43, for example, rats, hamsters, guinea pigs, rabbits, etc. can also be analyzed. Additionally, various animals such as pigs, cows, sheep, chickens, etc., and even dogs, cats, monkeys, humans, etc. can be targeted. In the above-described embodiment, for each of the plurality of unit images, parts of the bodies of one or more animals are extracted, the extracted plurality of parts are analyzed for each time series, and based on the analysis results, the individuals of one or more animals in each of the plurality of unit images are identified. However, this order is just an example. Additionally, for example, after individual recognition (identification), single parts may be extracted. Also, individual identification and part extraction may be performed simultaneously. When extracting single parts after individual recognition (identification), first, it is determined where each part is located in the image, trimmed, and then the single parts are extracted. Also, when identifying individuals after detecting the parts of multiple individuals, multiple detections are made of where specific parts are in the image, and then it is determined to which set (in this case, the set of parts of a specific individual) each part belongs based on speculation about "how their positions change over time". That is, a procedure may be included in which the position of the part of the animal is known, and in addition, the individual is identified, and as a result, the social nature of the individual is known.

[0138] Hereinafter, with reference to FIGS. 31 to 54, various indexes (for each of the initial movement period immediately after putting the mouse into the gauge and the period until it stabilizes, movement trajectory, distribution of existence positions for each area, existence time for each area, movement distance, total movement amount, body orientation, speed, angular velocity, etc.) are used to show the position, center of gravity position, movement range when the outline (contour) of a part (such as eyes and nose) of the body of a mouse (animal) extracted from a video (a plurality of unit images arranged in the order of the time axis) changes over time, the amount of movement, habits of behavior, rhythm, etc. Note that the graphs described below show a pair of a graph from the start of measurement to 1320 FRAME (44 seconds) of the initial movement and a graph from the start of measurement to 18030 FRAME (about 10 minutes).

[0139] First, with reference to FIGS. 31 and 32, the relationship between the position of the mouse in the cage and its movement trajectory will be described. FIG. 31 is a graph showing the position of the mouse existing in the cage and its movement trajectory from the start of measurement to 1320 FRAME (44 seconds) of the initial movement. FIG. 32 is a graph when the graph of FIG. 31 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0140] The vertical axis of FIGS. 31 and 32 indicates the length in the depth direction of the floor surface of the cage, and the vertical axis indicates the length in the width direction of the floor surface of the cage in terms of the number of pixels (PIXCEL). In the graph of 1320 FRAME of the initial movement in FIG. 31, the mouse is moving along the edge, avoiding the center of the cage, suggesting that it is anxious. In the graph from the start of measurement to 10 minutes in Fig. 32, it can be seen that there are still more individuals moving on the edge side than in the central part of the cage. On the other hand, moving lines indicating linear movement and parts where the moving lines are entangled can also be observed. In the parts where the moving lines are entangled, it can be seen that the mouse stays in that position and performs some kind of behavior (such as grooming). By having AI learn this information, it is possible to identify the behavior and mental disorders of the mouse, etc.

[0141] Next, referring to Figs. 33 and 34, the distribution of the presence of mice in each area obtained by dividing the floor of the cage into nine areas will be described. The vertical and horizontal axes in Figs. 33 and 34 indicate pixels (PIXCEL). Fig. 33 is a graph showing the distribution of the presence positions of mice in each of the nine areas (numbered 0 to 8) of the floor of the cage from the start of measurement to the first 1320 FRAME (44 seconds) of the initial movement. Fig. 34 is a graph when the graph of Fig. 33 is continuously drawn until 18030 FRAME (about 10 minutes) later.

[0142] Figs. 33 and 34 are graphs in which the positions of the mice in the cage are plotted every second. By attaching the identification information of the area including the coordinates to the coordinate information of the positions where the mice exist on the floor of the cage divided into nine areas and managing them, the graphs of Figs. 33 and 34 are created. In the graph of the first 1320 FRAME of Fig. 33, the number of plots is large in the areas near the edges of the cage (such as areas 0, 5, 6, 8, etc.), and it can be seen that the mice spend a lot of time near the edges. After that, in the graph of Fig. 34 continuously drawn until about 10 minutes, although the mice are still mostly near the edges, the number of plots in area 4 in the central part has also increased, and it can be seen that the frequency of the mice moving to the central part of the cage has increased. It can be said that the mouse's mood has become calmer than at the start of the initial movement.

[0143] Next, referring to Figs. 35 and 36, the presence time of the mice in each of the above-mentioned areas (0 to 8) will be described. FIG. 35 is a bar graph showing the presence time of the mouse in each of the nine areas into which the floor surface of the cage is divided from the start of measurement to 1320 FRAME (44 seconds) of the first motion. FIG. 36 is a bar graph when the graph of FIG. 35 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0144] In FIGS. 35 and 36, the vertical axis represents the number of plots (COUNTS), and the horizontal axis represents the region (REGION). The numerical values on the horizontal axis correspond to the numbers (0 to 8) of each area in FIG. 33. FIGS. 35 and 36 are graphs that count the number of plots where the mouse is present for each area, that is, graphs that accumulate the presence of the mouse for each area. When one plot (dot) in FIGS. 33 and 34 increases, the bar graph of the corresponding numbered area in FIGS. 35 and 36 becomes one count higher. From FIGS. 33 to 36, it can be visually (at a glance) seen in which areas of the gauge the mouse stays more (or less).

[0145] Next, with reference to FIGS. 37 and 38, the presence time of the mouse near the edge and near the center of the cage will be described. FIG. 37 is a graph showing the presence time of the mouse when divided into near the center (area 4 shown in FIG. 33) and near the edge (areas other than area 4) of the cage from the start of measurement to 1320 FRAME (44 seconds) of the first motion. FIG. 38 is a graph when the graph of FIG. 37 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0146] In FIGS. 37 and 38, the vertical axis represents the number of plots (COUNTS), and the horizontal axis represents the region (REGION). The region (REGION) is divided into two parts: near the center (area 4) and near the edge (areas other than area 4). In the graph at the start of the first motion in FIG. 37, it can be seen that the mouse is often near the edge (areas other than area 4). In the graph of FIG. 38, although it is overwhelmingly near the edge (areas other than area 4), it can be seen that the frequency of the mouse being near the center (areas other than area 4) is higher than at the start of the first motion.

[0147] Next, referring to FIGS. 39 and 40, the moving distance of the mouse per second (30 FRAMES) will be described. FIG. 39 is a graph showing the relationship between the period of the first 1320 FRAMES from the start of measurement and the moving distance of the mouse per second (30 FRAMES). FIG. 40 is a graph when the graph of FIG. 39 is continuously drawn until 18030 FRAMES (about 10 minutes) thereafter.

[0148] In FIGS. 39 and 40, the vertical axis represents distance (PICXEL), and the horizontal axis represents time (FRAME). The graphs in FIGS. 39 and 40 show the length (PICXEL on the image) between the place the mouse has passed and the place it is now at every 30 frames, that is, every second. That is, the graphs in FIGS. 39 and 40 show the moving distance of the mouse per second. The higher the oscillation in the vertical direction, the longer the moving distance in a short period, and the closer the value on the vertical axis is to 0, the more it indicates that the mouse stays in place. Also, in each graph of FIGS. 39 and 40, the broken line presented near the center of each oscillation amplitude indicates the moving average. In this way, by taking the average value within a certain width of time, the moving speed of the mouse can be understood.

[0149] Since these graphs are time-series graphs, for example, as in the graph of FIG. 40, when the mouse calms down in the latter half, the movement decreases. Among them, a mouse that continues to move at a constant speed can be said to be calm. On the other hand, a hypothesis can be established that a mouse that suddenly starts running or has an abnormally long stop time has an abnormality in its mental state.

[0150] Next, referring to FIGS. 41 and 42, the orientation of the mouse's body in the cage will be described. FIG. 41 is a graph showing the relationship between time at the first 1320 FRAMES from the start of measurement and the moment-by-moment orientation of the mouse's body in the cage. FIG. 42 is a graph when the graph of FIG. 41 is continuously drawn until 18030 FRAMES (about 10 minutes) thereafter.

[0151] In FIGS. 39 and 40, the vertical axis represents angle (DEGREE), and the horizontal axis represents time (FRAME). In the graphs of FIGS. 39 and 40, when the orientation of the mouse's body at any arbitrary time is set to straight (0 degrees on the vertical axis of the graph), and from that point on, for example, if the mouse faces right, the graphs in FIGS. 41 and 42 will swing in the negative direction. For example, if the mouse turns backward, etc., since the body orientation changes by more than 150 degrees, the graph will swing violently.

[0152] Next, referring to FIGS. 43 and 44, the total movement distance (total movement amount) of the mouse within the cage will be described. FIG. 43 is a graph showing the relationship between the time from the start of measurement to the first 1320 FRAME (44 seconds) of movement and the total movement distance of the mouse. FIG. 44 is a graph when the graph of FIG. 43 is continuously drawn until 18030 FRAME (about 10 minutes) later.

[0153] In FIGS. 43 and 44, the vertical axis represents distance (PICXEL), and the horizontal axis represents time (FRAME). The graph in FIG. 43 is the total distance (total movement distance), which is a graph obtained by stacking up the above-described movement distances. Starting from the measurement start point as 0, the distance the mouse has moved is added up, and finally, the movement distance from the start of measurement is shown. In the graphs of FIGS. 43 and 44, in the graph of the above-described movement speed (the graph of FIG. 39), it was difficult to understand the state at each moment, but by comparing this graph of the total movement distance with the graph of the movement speed, it becomes possible to understand things such as whether it has stopped at that moment. In this graph, it can be seen that the movement distance is finally more or less, but it can also be seen at which intermediate timing the movement amount suddenly increases. That is, with these graphs, the behavior of the mouse can be grasped from both the aspect of speed and the aspect of the movement amount. For example, in the graph of FIG. 37, the first red point (round point in the figure) oscillates up and down over time. However, when the red point moves horizontally, it indicates that the mouse is not moving, and the value approaches 0 in the speed graph of FIG. 37.

[0154] Next, referring to FIGS. 45 and 46, the behavior of the mouse rotating within the cage will be described. FIG. 45 is a graph showing the relationship between the time from the start of measurement to the first 1320 FRAME (44 seconds) and the rotational behavior (angular velocity) of the mouse. FIG. 46 is a graph when the graph of FIG. 45 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0155] In FIGS. 45 and 46, the vertical axis indicates the distance X angle (PIXEL × RAD), and the horizontal axis indicates the time (FRAME). The graphs of FIGS. 45 and 46 are graphs obtained by graphing the change in the body's orientation and the angle of the body's orientation over time series, and are obtained by multiplying the moving distance by the angle of the body's orientation change, such as 20 degrees, 30 degrees, etc. In the graphs of FIGS. 45 and 46, if it simply stays in place and rotates, the value will be 0, and if it moves while changing the body's orientation, the graph will fluctuate greatly up and down. From the graphs of FIGS. 45 and 46, for example, if only the body's orientation changes, a behavior of looking around can be recognized. However, if it stops in place and rotates until it changes the body's orientation, a mental state of desperately trying to check the surrounding situation can be inferred. By considering not only the comparison of the mouse's behavior with the immediately previous body orientation but also the movement, different behaviors can be seen.

[0156] Next, referring to FIGS. 47 and 48, the speed at which the mouse moves within the cage will be described. FIG. 47 is a graph showing the relationship between the time from the start of measurement to the first 1320 FRAME (44 seconds) and the speed at which the mouse moves. FIG. 48 is a graph when the graph of FIG. 47 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0157] In FIGS. 47 and 48, the vertical axis indicates distance / time (PIXEL / SEC), and the horizontal axis indicates time (FRAME). The graphs in FIGS. 47 and 48 are graphs with a concept similar to the graphs of the moving speeds in FIGS. 39 and 40. The graphs in FIGS. 39 and 40 measure the distance every 30 frames. For example, although data also exists between the 0th frame, the 30th frame, and the 60th frame, in the graphs of FIGS. 39 and 40, the moving speed was calculated by comparing only the 0th frame and the 30th frame, and the 30th frame and the 60th frame. However, in the graphs of FIGS. 47 and 48, the speed is calculated by taking the difference between the 0th frame and the 1st frame and dividing it by the time, which can be said to be a more detailed speed index. The speed of time passage for a mouse and a human is different, and 1 second for a human corresponds to 0.something seconds for a mouse. The graphs in FIGS. 39 and 40 output the speed per second. However, for the sake of the human sense, in the graphs of FIGS. 47 and 48, by outputting the speed with a finer mouse sense, the behavior of the mouse can be more represented.

[0158] Next, with reference to FIGS. 49 and 50, the angular velocity at which the mouse rotates within the cage will be described. FIG. 49 is a graph showing the relationship between the time from the start of measurement to the first 1320 FRAME (44 seconds) and the angular velocity at which the mouse moves. FIG. 50 is a graph when the graph of FIG. 49 is continuously drawn until 18030 FRAME (about 10 minutes) thereafter.

[0159] In FIGS. 49 and 50, the vertical axis indicates angle / time (DEGREE×SEC), and the horizontal axis indicates time (FRAME). The graphs in FIGS. 45 and 46 showed the relationship between the orientation and distance of the mouse's body, while the graphs in FIGS. 49 and 50 are calculated by dividing the numerical values of the graphs in FIGS. 45 and 46 by time. In the graphs of FIGS. 45 and 46, the changes in the angle of the mouse's body were quantified. However, the graphs of FIGS. 49 and 50 are the time derivatives of the angle changes. Similar to the relationship between the speed and the moving distance (amount of movement) described above, they represent the relationship between the angle and the angular velocity. Therefore, by comparing them with the graphs of FIGS. 45 and 46, it is possible to verify, for example, whether the mouse is stopped at a certain moment. That is, in the graphs of FIGS. 45 and 46, the moving speed and the amount of movement can be viewed simultaneously. For example, if there are numerical values of jumping up at each moment, the higher the height of the jump, the more it can be understood that the mouse is rotating and moving quickly. On the other hand, the lower the numerical value (the closer the numerical value is to 0), the slower it can be said that the mouse is rotating and moving.

[0160] Subsequently, with reference to FIGS. 51 to 54, operations including the detection of the outline of the mouse and the detection of the orientation of the body from the video will be described. FIG. 51 is a diagram showing a unit image at a certain timing of a video (original video) obtained by imaging inside the gauge with a camera. FIG. 52 is a diagram showing the outlines (contours) of a plurality (two) of mice identified from the unit image. FIG. 53 is a diagram showing the orientation (direction from the center of the body to the tip of the nose) of one of the two mice. FIG. 54 is a diagram showing the orientation (direction from the center of the body to the tip of the nose) of the other of the two mice.

[0161] In the above-described first and second embodiments, examples of separately performing the process of detecting the outline (contour) and the process of extracting the body parts have been described. However, by combining these processes, the behavior of the mouse can be analyzed in more detail.

[0162] In this case, at a certain timing of the original video, the unit image shown in FIG. 51 is extracted, the extracted unit image is subjected to image analysis, and after individual identification, as shown in FIG. 52, the outlines (contours) of a plurality (two) of white mice are color-coded and output, for example, in red and blue. Also, as shown in FIG. 53, the center of gravity of one of the two mice (the mouse with the upper outline (red) in FIG. 52) is detected, and the orientation of the body is detected from the positional relationship of the ears, eyes, and nose tip. By drawing a line segment from the center of gravity in the direction of the nose tip, the orientation of the mouse's face (the direction of the nose tip) is detected.

[0163] Similarly, as shown in FIG. 54, the center of gravity of the other one of the two mice (the mouse with the lower outline (blue) in FIG. 52) is detected, and the orientation of the body is detected from the positional relationship of the ears, eyes, and nose tip. By drawing a line segment from the center of gravity to the nose tip, the orientation of the mouse's face (the direction of the nose tip) is detected.

[0164] In this way, by combining the respective processes of outer contour identification and part extraction, the orientation of the mouse's face (the direction of the nose tip) can be detected, and the actions of the mouse such as rotational movement, forward movement, and backward movement in a state of staying in place can be analyzed in more detail.

[0165] In other words, the information processing apparatus to which the present invention is applied can take various embodiments having the following configurations. The information processing apparatus corresponds to the image processing apparatus 2 having the functional configuration of FIG. 16 described in the second embodiment and the server 200 having the functional configuration of FIG. 24 described in the third embodiment. That is, the second information processing apparatus (for example, the image processing apparatus 2 in FIG. 16, the server 200 in FIG. 24, etc.) to which the present invention is applied image acquisition means (for example, the image acquisition unit 51 in FIG. 16) for acquiring an analysis target image (for example, a video such as a moving image) configured by arranging a plurality of unit images in the time direction, which is obtained as a result of imaging the state in which one or more animals are active within a certain action range; outline detection means (for example, the outline detection unit 57 in FIG. 16) for detecting the outer outline (body contour, etc.) of each of the one or more animals using a skeleton estimation model that estimates and outputs the skeleton of the animal's body when the unit image is input for each of the plurality of unit images; For an individual identification model that outputs an individual of the animal when inputting the time series of one or more outlines of the body of the animal, based on the output obtained by inputting the time series of the outline (such as the contour of the body) of each of the one or more animals detected from each of the plurality of unit images by the outline detection means (for example, the outline detection unit 57 in FIG. 16), individual identification means (for example, the individual identification unit 54 in FIG. 16) for identifying the individual of each of the one or more animals in each of the plurality of unit images, Designation means (for example, the designation unit 471 in FIG. 24, etc.) for designating analysis attributes (type of analysis target (for example, mouse, rat, etc.), imaging direction of the animal to be analyzed (for example, top, obliquely upward, horizontal, etc.), color of the animal to be analyzed (for example, white, black, etc.)) for the image to be analyzed, Model selection means (for example, the model selection unit 472 in FIG. 24, etc.) for selecting a target to be applied to the outline detection means (for example, the outline detection unit 57 in FIG. 16) from among a plurality of types of the skeleton estimation models and selecting a target to be applied to the individual identification means (for example, the individual identification unit 54 in FIG. 16) from among a plurality of types of the individual identification models based on the analysis attributes of the image designated by the designation means, is provided. In the information processing apparatus configured in this way, a plurality of types of individual identification models and individual identification models are prepared in advance. When the analysis attributes for the image to be analyzed are designated, based on the analysis attributes of the designated image, a target to be applied to the outline detection means (for example, the outline detection unit 57 in FIG. 16) is selected from among a plurality of types of skeleton estimation models, and a target to be applied to the individual identification means (for example, the individual identification unit 54 in FIG. 16) is selected from among a plurality of types of individual identification models. Then, when there is an analysis instruction for the image, using the selected animal's skeleton estimation model and individual identification model, a plurality of outlines extracted from each of the plurality of unit images of the image are analyzed for each time series, and based on the analysis results, the individuals of one or more animals are identified. Therefore, the individuals of each mouse can be correctly distinguished from an image of one or more animals (for example, mice, etc.) moving within a certain activity range. Furthermore, the second information processing apparatus (for example, the image processing apparatus 2 in FIG. 16, the server 200 in FIG. 24, etc.) generating means (for example, the data frame generation unit 53 in FIG. 16) that generates a data frame indicating the probability that the outer contour that changes with the behavior of the animal is a specific behavior; behavior determination means (for example, the behavior determination unit 56 in FIG. 16) that determines an action corresponding to the satisfied condition when the value of the probability of the specific action in the data frame satisfies any one of one or more preset conditions; further comprising. With this configuration, when the value of the probability that the outer contour of the mouse with changes is a specific behavior satisfies the condition, the behavior corresponding to the condition is determined, so that the reason for the behavior of each mouse moving within a certain behavior range can be known. The behavior determination means (for example, the behavior determination unit 56 in FIG. 16) behavior detection means (for example, the behavior detection unit 81 in FIG. 16) that detects a behavior related to the positional relationship (such as scratching behavior or grooming) when a condition defining the positional relationship between the outer contour of the animal and other parts (outer contours of other animals or corners of the cage) is satisfied; further comprising This makes it possible to know the reasons for behaviors such as being stressed due to scratching behavior while the animal is moving within a limited behavior range. Also, it is possible to know the reasons for behaviors such as trying to have a relationship with other animals by performing grooming behavior on other animals. The behavior includes at least one of behaviors related to the sociality of each of the animals, behaviors related to the interaction between animals existing in the same behavior range, and behaviors related to the relationship between animals. Thereby, it is possible to know whether the reason for the behavior of the animal is a behavior related to sociality, a behavior related to the interaction between animals existing in the same behavior range, or a behavior related to the relationship between animals. A marker image generation means (for example, the marker image generation unit 55 in FIG. 16) that generates a marked image in which a marker indicating the outline of each body of the animal detected by the outer outline detection means (for example, the outer outline detection unit 57 in FIG. 16) and the individual of the animal identified by the individual identification means (for example, the individual identification unit 54 in FIG. 16) is associated with the outer outline. is provided. Thus, by displaying the marked image, one or more animals existing in the image can be distinguished by the marker. In addition, after distinguishing the animals, the outlines of those individuals can be detected, and significance can be assigned to their changes (body movements) for evaluation. The second information processing device (for example, the image processing device 2 in FIG. 16, the server 200 in FIG. 24, etc.) An image acquisition means (for example, the image acquisition unit 51 in FIG. 16) that acquires a group of images (video) in which frame images composed of a plurality of unit images (pixels) obtained as a result of imaging the activities of one or more animals within a certain action range are arranged in the time direction. A determination means (such as the individual recognition unit 91 in FIG. 16) that determines to which region of each of the one or more animals each of the unit images belongs. A specifying means (for example, the data frame generation unit 53 in FIG. 16) that specifies the position of the outer outline (such as the body contour) of each animal in coordinates indicating the distance from a predetermined reference point in the image for each region of the animal. An individual identification means (for example, the individual identification unit 54 in FIG. 16) that analyzes how the position and range of the outer outline specified by the specifying means change over time, and identifies (classifies) each of the one or more animals included in the image based on the analysis result. is provided. In addition, for each animal identified (classified) by the individual identification means (for example, the individual identification unit 54 in FIG. 16), it is determined whether the change in the outer outline matches a preset condition, and an action label (scratching action, sleeping, grooming action, etc.) of the animal corresponding to the matched condition is assigned to the time information (timestamp) of the image including the part by an action determination means (for example, the action determination unit 56 in FIG. 16). is further provided. As a result, it is possible to know which behavior among scratching behavior, during sleep, grooming behavior, etc. the behavior of the animal belongs to.

Explanation of symbols

[0166] 1 ··· Camera, 2 ··· Information processing device, 11 ··· CPU, 41 ··· Image DB, 42 ··· Model DB, 43 ··· Material DB, 51 ··· Image acquisition unit, 52 ··· Part extraction unit, 53 ··· Data frame generation unit, 54 ··· Individual identification unit, 55 ··· Marker image generation unit, 56 ··· Behavior determination unit, 57 ··· Outline detection unit, 61 ··· Video acquisition unit, 62 ··· Unit image generation unit, 81 ··· Behavior detection unit, 82 ··· Behavior prediction unit, 71, 91 ··· Individual recognition unit, 72 ··· Part detection unit, 92 ··· Outline specification unit, 450 ··· Image analysis unit, 451 ··· Web service unit, 452 ··· Process unit, 461 ··· Authentication unit, 462 ··· Search unit, 463 ··· Search result output unit, 464 ··· Analysis data addition unit, 465 ··· Analysis result display control unit, 471 ··· Designation unit, 472 ··· Model selection unit, 491 ··· Upload data management unit, 492 ··· Unprocessed data management unit

Claims

1. an image acquisition means for acquiring an image to be analyzed, the image being obtained by capturing an image of one or more animals moving within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; an outline detection means for detecting an outline of a body of each of the one or more animals by using a skeleton estimation model which estimates and outputs an animal's skeleton when a unit image is input, for each of the plurality of unit images; an individual identification means for identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the outline of one or more of the bodies of the animals detected from each of the plurality of unit images by the outline detection means into an individual identification model which outputs an individual of the animal when a time series of the outline of the body of the animal is input; A designation means for designating an analysis attribute for the image to be analyzed; a model selection means for selecting an object to be applied to the contour detection means from among a plurality of types of the skeleton estimation models based on the analysis attribute of the image designated by the designation means, and for selecting an object to be applied to the individual identification means from a plurality of types of the individual identification models; a generating means for generating a data frame indicating a transition of the position of the outer shell which changes in accordance with the behavior of the animal; and a behavior determination means for determining, when a transition in the position of the outer boundary in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, The information processing device, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

2. The behavior determination means a behavior detection means for detecting a behavior related to a positional relationship between the outer shell of the animal and another specific part when a condition that specifies the positional relationship is satisfied; The information processing device according to claim 1 , further comprising:

3. The action, At least one of the following behaviors is included: behaviors related to the sociality of each animal, behaviors related to interactions between animals in the same home range, and behaviors related to relationships between animals; The information processing device according to claim 2 .

4. An information processing method executed by an information processing device, an image acquiring step of acquiring an image obtained by capturing an image of one or more animals being active within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a contour detection step of detecting a contour of each of the one or more animals using a skeleton estimation model that estimates and outputs a skeleton of an animal's body when a unit image is input for each of the plurality of unit images; an individual identification step of identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the outer contour of one or more bodies of the animal to an individual identification model which outputs an individual of the animal when a time series of the outer contour of the body of the animal is input; A designation step of designating an analysis attribute for the image to be analyzed; a model selection step of selecting an object to be applied to the contour detection step from among a plurality of types of the skeleton estimation models, and selecting an object to be applied to the individual identification step from among a plurality of types of the individual identification models, based on the analysis attributes of the specified image; A generation step of generating a data frame showing a transition of a position of a part that changes according to the behavior of the animal; and a behavior determination step of determining, when a transition of the position of the body part in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, An information processing method, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

5. A computer that controls an information processing device includes: an image acquiring step of acquiring an image obtained by capturing an image of one or more animals being active within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a contour detection step of extracting a contour of each of the one or more animals' bodies using a skeleton estimation model that estimates and outputs a skeleton of an animal's body when a unit image is input, for each of the plurality of unit images; an individual identification step of identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the outer contour of one or more bodies of the animal to an individual identification model which outputs an individual of the animal when a time series of the outer contour of the body of the animal is input; A designation step of designating an analysis attribute for the image to be analyzed; a model selection step of selecting an object to be applied to the contour detection step from among a plurality of types of the skeleton estimation models, and selecting an object to be applied to the individual identification step from among a plurality of types of the individual identification models, based on the analysis attributes of the specified image; A generation step of generating a data frame showing a transition of a position of a part that changes according to the behavior of the animal; and a behavior determination step of determining, when a transition of the position of the body part in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, A program for executing a control process, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

Images