Information processing device, information processing method, and program

Abstract

The present invention addresses the problem of making it possible to identify respective individuals of mice from an image taken of at least one mouse in action within a certain behavior range.　An image processing device 2 is provided with an image acquisition unit 51, a part extraction unit 52, and an individual identification unit 54. The image acquisition unit 51 acquires a video obtained as the result of imaging at least one animal in action within a certain behavior range. The part extraction unit 52 extracts a part of the body of each of the at least one animal, from each of a plurality of unit images included in the video. The individual identification unit 54 analyzes the plurality of parts extracted from each of the plurality of unit images by the part extraction unit 52 in chronological order, and, on the basis of the analysis result, identifies each individual of the at least one animal from each of the plurality of unit images. The problem is thus solved.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art As a conventional technique, there is a technique for identifying an individual animal such as a cat while processing the individual's biometric information (for example, Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-007625 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the conventional techniques described in the above documents are capable of measuring an animal's biological data in a non-contact manner, they are not capable of identifying an animal that is constantly moving or an individual animal when multiple animals are present. In addition, the conventional technology requires a thermograph and the like in addition to a camera to check the health condition of an individually identified animal when processing the biometric information of an individual, which is costly. Furthermore, the above document does not mention any uses other than health condition management.

[0005] The present invention has been made in view of these circumstances, and has as its object to make it possible to distinguish one or more individual mice moving within a certain behavioral range. [Means for solving the problem]

[0006] In order to achieve the above object, an information processing device according to one aspect of the present invention comprises: an image acquisition means for acquiring an image to be analyzed, the image being obtained by capturing an image of one or more animals moving within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a part extraction means for extracting a part of a body of each of the one or more animals using a skeleton estimation model which estimates and outputs a skeleton of an animal's body when a unit image is input, for each of the plurality of unit images; an individual identification means for identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the one or more body parts of the animal extracted from each of the plurality of unit images by the part extraction means into an individual identification model which outputs an individual of the animal when a time series of the one or more body parts of the animal is input; A designation means for designating an analysis attribute for the image to be analyzed; a model selection means for selecting a model to be applied to the part extraction means from among a plurality of types of the skeletal estimation models based on the analysis attribute of the image designated by the designation means, and for selecting a model to be applied to the individual identification means from among a plurality of types of the individual identification models; Equipped with. In this way, multiple types of skeletal estimation models and multiple types of individual identification models are prepared, and based on the analysis attributes of the image specified by the specification means, a skeletal estimation model to be applied to the part extraction means is selected from the multiple types of skeletal estimation models, and an individual identification model to be applied to the individual identification means is selected from the multiple types of individual identification models. Then, in response to an analysis instruction, body parts of each of the one or more animals are detected for each of the multiple unit images contained in the image, the detected body parts are analyzed in chronological order, and each of the one or more animals is identified based on the analysis results, making it possible to distinguish individual animals from images capturing one or more animals active within a certain range of movement.

[0007] An information processing method and a program corresponding to the information processing device according to one aspect of the present invention are also provided as an information processing method and a program according to one aspect of the present invention. Effect of the Invention

[0008] According to the present invention, it is possible to distinguish one or more individual mice that are active within a certain behavioral range. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system including an image processing device according to an embodiment of the information processing device of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of an image processing device according to an information processing device of the present invention, in the information processing system of FIG. [Diagram 3] 1; FIG. 3 is a functional block diagram showing a first embodiment of the information processing system in FIG. 1, that is, a first embodiment of the functional configuration of the image processing device in FIG. 2; [Figure 4] 4 is a flowchart showing the operation of an image processing device having the functional configuration of FIG. 3. [Diagram 5] FIG. 2 is a diagram showing an example of a unit image acquired from a video. [Figure 6] 6 is a diagram showing an example of a unit image acquired from a video after the unit image in FIG. 5. FIG. [Figure 7] 7 is a diagram showing an example of a unit image acquired from a video after the unit image in FIG. 6; FIG. [Figure 8] FIG. 6 is a diagram showing a state in which individuals are identified from the unit image in FIG. 5. [Figure 9] 9 is a diagram showing a state in which parts are detected from the unit image in FIG. 8. FIG. [Figure 10] FIG. 10 is a diagram showing a state in which the parts in FIG. 9 are extracted. [Figure 11] 8 is a diagram showing how the parts extracted from the unit images of FIGS. 5 to 7 are tracked. FIG. [Figure 12] 8 is a diagram showing a tracking image in which a marker is added to a unit image at time t3 in FIG. 7. FIG. [Figure 13] FIG. 13 is a diagram showing an example of detection of water drinking behavior of a mouse. [Figure 14] FIG. 13 is a diagram showing an example of detection of feeding behavior of a mouse. [Figure 15] FIG. 13 is a diagram showing an example of detection of mutual interference behavior between mice. [Figure 16] 1, that is, a functional block diagram showing a second embodiment of the functional configuration of the image processing device in FIG. 2. FIG. [Figure 17] 17 is a flowchart showing the operation of an image processing device having the functional configuration of FIG. 16. [Figure 18] FIG. 2 is a diagram showing an example of a unit image acquired from a video. [Figure 19] 19 is a diagram showing an example of a unit image acquired from a video after the unit image in FIG. 18. FIG. [Figure 20] FIG. 20 is a diagram showing how a contour is detected from the unit image in FIG. 18. [Figure 21] FIG. 20 is a diagram showing how a contour is detected from the unit image in FIG. 19. [Figure 22] FIG. 13 is a diagram showing an example of detection of a scratching behavior. [Figure 23] FIG. 2 is a diagram showing an outline of a business model for commercially using the information processing system of FIG. 1. [Figure 24] 1, that is, a functional block diagram showing the functional configuration of a server in which the functional configuration of the image processing device in FIG. 3 is used as an image analysis unit, and a client-side PC. [Diagram 25] FIG. 25 is a diagram showing a search screen displayed on the PC in FIG. 24. [Figure 26] FIG. 26 is a diagram showing an analysis data addition screen which is displayed as a pop-up on the search screen of FIG. 25. [Figure 27] FIG. 26 is a diagram showing a video confirmation screen that is displayed as a pop-up on the search screen of FIG. 25. [Figure 28] FIG. 13 is a diagram showing an example of a report created from a video. [Figure 29] FIG. 1 is a diagram showing an example of a configuration of a closed network connection that provides robust security. [Diagram 30] FIG. 2 is a diagram showing a dashboard screen of the information processing system. [Diagram 31] This is a graph showing the position of the mouse in the cage and its movement trajectory from the start of measurement until the initial 1320 frame (44 seconds). [Diagram 32] This is a graph of the graph in Figure 31 when it was continuously drawn up to 18030 FRAMEs (about 10 minutes). [Diagram 33] The cage floor was divided into nine areas (0 to 8) and the graph shows the distribution of the mouse's position for each of these areas from the start of measurement until the initial 1320 frame (44 seconds). [Diagram 34] This is a graph of the graph in Figure 33 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Diagram 35] This is a bar graph showing the time the mouse spent in each of the nine areas on the cage floor, from the start of measurement until the initial 1320 frame (44 seconds). [Diagram 36] This is a bar graph of the graph in Figure 35, which was drawn continuously up to 18030 FRAME (approximately 10 minutes). [Figure 37] This is a graph showing the time the mouse spent in the cage, divided into near the center (area 4) and other areas (near the edges) from the start of measurement until the initial 1320 frame (44 seconds). [Figure 38] This is a graph of the graph in Figure 37 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Figure 39] This is a graph showing the relationship between the period of 1320 frames from the start of measurement and the distance the mouse moved per second (30 frames). [Diagram 40] This is a graph of the graph in Figure 39 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Diagram 41] This is a graph showing the relationship between the period of 1320 frames from the start of measurement to the initial movement and the moment-to-moment body orientation of the mouse in the cage. [Diagram 42] This is a graph of the graph in Figure 41 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Diagram 43] 13 is a graph showing the relationship between the time from the start of measurement to the initial 1320 FRAME (44 seconds) and the total distance (total amount) moved by the mouse. [Diagram 44]This is a graph of the graph in Figure 43 when it was continuously drawn up to 18030 FRAMEs (about 10 minutes). [Diagram 45] 1 is a graph showing the relationship between the time from the start of measurement to the initial 1320 FRAME (44 seconds) and the rotational behavior (angular velocity) of the mouse. [Figure 46] This is a graph of the graph in Figure 45 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Figure 47] This is a graph showing the relationship between the time from the start of measurement to the initial 1320 FRAME (44 seconds) and the speed of mouse movement. [Figure 48] This is a graph of the graph in Figure 47 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Figure 49] This is a graph showing the relationship between the time from the start of measurement to the initial 1320 FRAME (44 seconds) and the angular velocity of mouse movement. [Figure 50] This is a graph of the graph in Figure 49 when it was continuously drawn up to 18030 FRAME (about 10 minutes). [Figure 51] FIG. 13 is a diagram showing unit images at a certain timing of a video (original video) obtained by capturing an image of the inside of a gauge with a camera. [Figure 52] FIG. 2 is a diagram showing the outlines (contours) of each of multiple mice (two mice) identified from a unit image. [Figure 53] FIG. 1 shows the orientation (direction from the center of the body to the tip of the nose) of one of two mice. [Figure 54] FIG. 13 shows the orientation (direction from the center of the body to the tip of the nose) of the other of the two mice. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing the configuration of an information processing system including an image processing device according to an embodiment of the information processing device of the present invention.

[0011] (First embodiment) The information processing system shown in Figure 1 is configured to include a camera 1 installed (positioned) so as to capture an image of a cage C housing one or more mice (mice X1 and X2 in the example of Figure 1) from above, and an image processing device 2 connected to the camera 1 via a network N. The network N includes wired networks as well as wireless networks. The cage C is a housing means for allowing an animal to move within a certain range of movement.

[0012] The camera 1 is, for example, a digital camera or a network camera that captures moving images, and outputs moving images captured inside the cage C from above to the image processing device 2. Here, a moving image refers to an image that is composed of a plurality of unit images arranged in the time direction, and is also called a video. Although field images may be used as the unit images, frame images are used here.

[0013] The image processing device 2 identifies the individual mice X1 and X2 included as subjects in the video by using the video acquired from the camera 1 and a learning model (details will be described later) stored in the model DB 42. The image processing device 2 further detects factors (causes) underlying the behavior of the mice X1 and X2, such as the sociality of the mice X1 and X2, the interaction between the mice X1 and X2 existing in the same behavioral range, and the relationship between the mice X1 and X2, from the behavioral patterns (habits, etc.) of the mice X1 and X2 identified as individuals. The image processing device 2 generates a tracking image with at least one of markers such as a part of the body skeleton (tracking point) or an individual ID (identification) of one or more mice contained in frame images extracted from the video, and an object for visually identifying the outer contour (outline from the surroundings) of each mouse, and outputs the tracking image to an output unit 16 such as a display. If a printer is connected to the image processing device 2 as the output unit 16, the tracking image can also be printed. The object is a contour line or a mask image showing the outline of the individual mouse, etc. The marker includes a tracking point shown by a figure such as a circle, triangle, or square, an individual ID shown by alphanumeric characters, etc. The functional configuration and processing of the image processing device 2 will be described in detail later with reference to FIG. 3 and subsequent figures.

[0014] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to the present invention in the information processing system of FIG.

[0015] The image processing device 2 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a memory unit 18, a communication unit 19, and a drive 20.

[0016] The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 18 into the RAM 13 . The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes.

[0017] The CPU 11, the ROM 12, and the RAM 13 are connected to one another via a bus 14. An input / output interface 15 is also connected to the bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15. The output unit 16 is composed of a display, a speaker, etc., and outputs images and sounds. The input unit 17 is composed of a keyboard, a mouse, etc., and inputs various information in response to user instructions. The storage unit 18 is configured with a hard disk or the like, and stores various types of information data.

[0018] The communication unit 19 controls communication with other communication targets (for example, the camera 1 in FIG. 1) via the network N. Removable media 21, which may be a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is appropriately attached to the drive 20. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as necessary. The removable media 21 can also store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0019] FIG. 3 is a functional block diagram showing a first embodiment of the information processing system in FIG. 1, that is, a first embodiment of the functional configuration of the image processing device 2 in FIG.

[0020] The storage unit 18 of the image processing device 2 shown in FIG.

[0021] The image DB41 stores data of a moving image acquired from the camera 1, data of a plurality of frame images (still image data) that make up the moving image, data of tracking images in which markers indicating objects or individuals for tracking animals are added to the frame images, and a table (data frame) showing the transition of positional information of the parts of the mice X1 and X2.

[0022] A plurality of learning models are stored in the model DB 42. Specifically, the model DB 42 stores a plurality of types (one or more) of skeletal estimation models for estimating the skeleton of each animal (mouse, rat, etc.), a plurality of types (one or more) of individual identification models for identifying each individual animal (if there are two of the same animal, each animal is identified as a different individual), a plurality of types (one or more) of mathematical models for determining the behavior of each individual animal, and the like. The skeleton estimation model is a learning model designed to output an animal's skeleton when an image of an animal is input. The individual identification model is a learning model designed to input a time series of one or more parts of an animal's body and output the individual animal. The mathematical model is a model for analyzing animal behavior that is designed to output information indicating what the animal's behavior is when one or more images of the animal changing over time are input. As a result of machine learning and updating of videos and still images of each individual animal prepared in advance, when a new video or still image is input, trained learning data is stored for identifying the animal from the input image, identifying the individual, and further analyzing the behavior of each individual and outputting the analysis results. Such trained learning data is referred to as a "learning model" in this specification.

[0023] Specifically, from a video obtained by filming the movement of a mouse, as an example of an animal, a pre-selected skeletal estimation model for a mouse estimates the arrangement of one or more bones (skeleton) of the mouse's head, neck, arms, legs, tail, etc., and a pre-selected individual identification model for a mouse identifies an individual from the positions and movements of feature points extracted from the animal's skeleton (body parts such as the mouse's eyes, nose, ears, toes, etc.), and the mathematical model outputs the results of the individual's behavior. In other words, the learning model in the first embodiment is a learning model that analyzes individual animals and their behavior using techniques of skeletal estimation, individual identification, and behavior estimation. Among the above models, for example, inside the skeleton estimation model, data of any pixel in the image (all pixels in the case of a model using CNN) is input and appropriate values ​​(coordinates of the nose position, probability of nose coordinates, etc.) are calculated according to the model for each animal. CNN refers to a neural network that is mainly composed of convolutional layers and pooling layers.

[0024] The learning model stored in the model DB42 is a reference learning model that is generated by a learning device (not shown) performing machine learning using one or more (a large number of) pieces of learning data in advance, and newly learned data is also added.

[0025] For example, a convolutional learning neural network or the like can be applied to the machine learning. Note that the convolutional learning neural network is merely an example, and other machine learning methods may be applied. Furthermore, the learning model is not limited to a machine learning model, and a classifier that identifies individual animals using a predetermined algorithm may be used. In other words, the learning model needs to be generated by learning so that when an image is input, if the skeleton, contour, and movements of each mouse contained in the image match those of a known mouse, the model will assign those attributes and output them.

[0026] The material DB 43 stores data that serves as materials for determining the reasons for mouse behaviors and sociality, etc., identified based on the learning model. The data in the material DB 43 associates mouse behaviors with the sociality derived from the behaviors and relationships with other mice, and can be used to determine various habits, quirks, ecology, etc., of mouse behaviors. For example, this data is not just about mouse movements, but can also be used to deduce habits such as whether a mouse that moves in a certain way is social or has a certain relationship with other mice. The material DB 43 stores data that associates conditions under which a mouse behaves with behavioral patterns of the mouse derived from the conditions. That is, the material DB 43 stores data that serves as a basis for the mathematical model to determine what kind of behavior the mouse detected from an image is. Examples of data that serves as a basis for the determination include behavioral data such as exploration, eating and drinking, running and walking, sleeping, scratching, grooming, and fighting.

[0027] As shown in FIG. 3, the image processing device 2 includes an image acquisition unit 51, a part extraction unit 52, a data frame generation unit 53, an individual identification unit 54, a marker image generation unit 55, a behavior determination unit 56, and the like.

[0028] The image acquisition unit 51 acquires images obtained by capturing images of one or more animals such as mice X1 and X2 moving within a certain range of movement.

[0029] The image acquisition unit 51 includes a moving image acquisition unit 61 and a unit image generation unit 62 . The video acquisition unit 61 acquires a video (video) captured by the camera 1. The unit image generation unit 62 generates an image to be analyzed, which is configured by arranging a plurality of unit images in the time direction, from the video (video) acquired by the video acquisition unit 61, and stores the image in the image DB 41. In other words, the unit image generation unit 62 generates a group (frame images) of a plurality of unit images (still images) in frame units from the video.

[0030] The part extraction unit 52 extracts body parts of one or more animals for each of a plurality of unit images using a skeleton estimation model selected from a plurality of types of skeleton estimation models in the model DB 42. That is, the part extraction unit 52 sequentially reads out a plurality of frame images in the image DB 41, and recognizes an image area of ​​an animal that is moving on its own from a unit image included in each frame image. That is, the part extraction unit 52 separates the background part of the frame image from the outline part of the animal's body and recognizes the image area of ​​the animal's body. A part of the body may be, for example, one or more of the following characteristic points: the left and right eyes, the nose, the left and right ears, the tips of the left and right front legs, the tips of the left and right hind legs, the tip of the tail, the junctions of bones or joints, and the center of gravity of the body. The unit images include the above-mentioned frame images, as well as one pixel, a group of multiple pixels, and the like.

[0031] The part extraction unit 52 includes an individual recognition unit 71 and a part detection unit 72 . The individual recognition unit 71 recognizes each of the unit images included in the frame image that has a change (movement) as part of one behaving individual. Specifically, the individual recognition unit 71 binarizes the unit images and recognizes the color different from the background image as part of the behaving individual. The individual recognition unit 71 regards the part where the color changes compared with the previous and next frame images in the time series as the boundary of the area. The part detection unit 72 extracts the body parts of the animal by superimposing the skeleton output from the skeleton estimation model on the area of ​​the object recognized as a part of the moving individual in each of the multiple unit images, and sets the body parts as tracking points. At least one or more body parts, such as the nose and the left and right eyes, are detected. In this embodiment, body parts of one or more animals are extracted from an image by combining a part extraction unit 52 equipped with an individual recognition unit 71 and a part detection unit 72 with a skeleton estimation model. However, a part extraction model that outputs the body parts of an animal when a unit image is input may be stored in advance in model DB 42, and the animal's body parts may be extracted by inputting the unit image into model DB 42.

[0032] The data frame generating unit 53 converts the body parts extracted by the part extracting unit 52 into a data frame. The data frame generating unit 53 identifies the positions of specific parts of the animal (eyes, nose, ears, legs, tail, skeleton, etc.) for each region of the animal using coordinates indicating the distance from a predetermined reference point in the image. Specifically, the data frame generating unit 53 generates a table that associates the body parts extracted by the part extracting unit 52 with position information that expresses the body parts in a two-dimensional (planar) coordinate system (x-axis, y-axis, etc.) based on a certain reference point of the cage C included in the frame image or part of the behaving individual. The generated table is called a data frame. The x-axis is the horizontal axis of the two-dimensional (planar) coordinate system, and the y-axis is the vertical axis of the plane. That is, the data frame generating unit 53 generates a data frame that indicates the transition of the position of the part that changes in accordance with the behavior of the animal. When acquiring a spatial position as a tracking point, a three-dimensional (solid) coordinate system (x-axis, y-axis, z-axis) is used. The z-axis is the depth axis of the three-dimensional (solid) coordinate system.

[0033] The individual identification unit 54 identifies each of the one or more animals in each of the multiple unit images based on the output obtained by inputting a time series of body parts of each of the one or more animals extracted from each of the multiple unit images by the part extraction unit 52 into an individual identification model selected from the multiple types of individual identification models in the model DB42. Specifically, the individual identification unit 54 uses an individual identification model selected from multiple types for the image to analyze one or more parts extracted from each of the multiple unit images by the part extraction unit 52 in time series, and identifies each of the one or more individual mice X1, X2 in each of the multiple unit images based on the analysis results. The individual identification section 54 analyzes how the position coordinates of the body parts change over time, and identifies (classifies as different individuals) each of the one or more animals included in the frame images based on the analysis results. In other words, individual identification unit 54 identifies what type of animal the one or more body parts belong to and which individual animal has those body parts.

[0034] Specifically, the individual identification unit 54 refers to the model DB 42, analyzes the data frames generated by the data frame generation unit 53 for each time series, identifies each of the one or more mice X1, X2 in each of the multiple data frames based on the analysis results, and assigns the individual IDs of the identified mice X1, X2 to tracking points.

[0035] The individual identification unit 54 converts, for example, the positional relationship of each part of each individual at any point in time and the frequency with which certain conditions are satisfied into data. Here, a certain condition means, for example, that the positions of a certain part and another part are within a certain coordinate position (specified range) for a certain period of time, or that the distance between a part of another individual is zero or close to zero for a certain period of time.

[0036] The individual identification unit 54 instructs the marker image generation unit 55 to generate an image in which at least one of markers indicating the tracking points of the mice X1 and X2 identified by the individual identification unit 54 and an object for visually identifying the individual mice X1 and X2 is superimposed on the unit image, and the resulting image with markers (tracking image) is output to the output unit 16.

[0037] The marker image generating unit 55 generates an image with markers in which the body parts of the mice X1 and X2 extracted by the part extracting unit 52 and the markers indicating the individuals of the mice X1 and X2 identified by the individual identifying unit 54 are associated with the body parts. Specifically, the marker image generating unit 55 generates a marker-attached image (tracking image) by attaching a marker to the unit image that enables visual identification of the individual mice X1 and X2 identified by the individual identification unit 54. The marker is, for example, an individual ID (character) or a tracking point for each individual. The object is a contour line, frame, mask image, etc., colored in a different color for each individual.

[0038] The behavior determination unit 56 determines the behavior of one or more mice X1, X2 identified by the individual identification unit 54. Here, the behavior includes not only the behavior of each of the mice X1, X2, but also the interaction behavior between the mice X1, X2 present in a unit image. That is, when the transition of the position of the body part in the data frame satisfies any one of one or more preset conditions, the behavior determination unit 56 determines the behavior according to the satisfied condition.

[0039] Specifically, the behavior determination unit 56 determines, for each animal identified (classified) by the individual identification unit 54, whether or not the change in the coordinates of the part matches a preset condition, based on a mathematical model selected from the multiple types of mathematical models in the model DB 42 and the material data in the material DB 43, and assigns the animal's behavior label corresponding to the matched condition (such as ``01'' for scratching behavior, ``02'' for sleeping, or ``03'' for grooming behavior) to the time information (time stamp) of the image frame or data frame containing the part.

[0040] As a result, for example, in an environment in which two mice X1 and X2 are housed and active in a cage C, if the frequency with which the nose (site) of one mouse X1 touches the nose (site) of the other mouse X2 exceeds a predetermined number of times within a certain period of time, it can be deduced that this is a grooming behavior in which the individuals are attempting to build a friendly relationship, or that in the case of a male or female, the individuals are attempting to engage in breeding behavior, etc.

[0041] The behavior determination unit 56 includes a behavior detection unit 81 and a behavior prediction unit 82 . When a condition that specifies a positional relationship between a part of an animal and another specific part (a water hole, a feeding area, a part of another animal) is satisfied, the behavior detection unit 81 detects a behavior related to the positional relationship. Specifically, the behavior detection unit 81 refers to the model DB 42 and the material DB 43, and checks whether the positional relationship between the parts of the mice X1 and X2 at any time in the data frame and the positional relationship between the mice X1 and X2 and other life-related members (such as a watering hole and a feeding hole) in the cage C satisfy the conditions in the material DB 43, and detects behavior corresponding to the satisfied conditions. At least one of the sociality of the mice X1 and X2, the interaction between the mice X1 and X2 that exist in the same behavioral range, and the relationship between the mice X1 and X2 is detected.

[0042] The behavior prediction unit 82 predicts, based on the relationship between the mice X1 and X2 detected by the behavior detection unit 81, how the mice X1 and X2 will live in the future.

[0043] Next, image processing executed by the information processing device will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating an example of the flow of image processing executed by the information processing device having the functional configuration of Fig. 3. In the information processing system of the first embodiment, one or more mice X1, X2 moving in a cage C are imaged by a camera 1, and the video is input to an image processing device 2, which then identifies each of the one or more mice X1, X2 and performs image processing to assist in determining the individuality, sociality, and the like of each of the mice X1, X2 as follows.

[0044] In step S11, the image acquisition unit 51 acquires an image (e.g., a video or the like) that is obtained by capturing an image of one or more animals being active within a certain movement range and that is composed of a plurality of unit images arranged in the time direction.

[0045] In step S12, part extraction unit 52 extracts a part of the body of each of one or more animals (eyes, nose, ears, feet, parts of bones or joints, center of gravity of the body, etc.) for each of multiple unit images included in the acquired video.

[0046] In step S13, the individual identification unit 54 analyzes the multiple parts extracted from each of the multiple unit images by the part extraction unit 52 in chronological order, and identifies each individual of one or more animals in each of the multiple unit images based on the analysis results.

[0047] Identifying individuals means, for example, that among mice X1 and X2 housed in cage C as shown in Figures 1 and 3, mouse X1 is identified as individual "0" by assigning an individual ID such as "0" to a part of the body, and mouse X2 is identified as individual "1" by assigning an individual ID such as "1" to a part of the body.

[0048] In step S14, the behavior determination unit 56 determines the behavior of one or more mice X1 and X2 identified by the individual identification unit 54.

[0049] In the behavior determination unit 56, the behavior detection unit 81 detects at least one of the sociality of each of the mice X1 and X2, the interaction between the mice X1 and X2 that are present in the same behavioral range, and the relationship between the mice X1 and X2, based on the positional relationship between the parts of the mice X1 and X2 at any time point.

[0050] Thus, according to the operation of the image processing device 2, a part of the body of each of the mice X1 and X2 is extracted from a moving image capturing the two mice X1 and X2 moving in the cage C, the extracted parts are analyzed in chronological order to identify each of the mice X1 and X2, and a tracking image is displayed in which markers such as individual IDs and tracking points or objects for individual identification are attached to each of the identified mice, so that each of the mice X1 and X2 existing and moving in the same image can be distinguished. In addition, as behaviors between mice X1 and X2, for example, the sociality of mice X1 and X2, interactions between mice X1 and X2 existing in the same behavioral range, and relationships between mice X1 and X2 are detected, so that the master-slave relationship (dominance, subordination, etc.), habits, ecology, etc. of mice X1 and X2 can be considered.

[0051] Next, the procedure of image processing by the image processing device 2 in this information processing system will be described with reference to FIGS. FIG. 5 is a diagram showing an example of a frame image acquired from a video. FIG. 6 is a diagram showing an example of a frame image acquired from a video after the frame image of FIG. 5. FIG. 7 is a diagram showing an example of a frame image acquired from a video after the frame image of FIG. 6. FIG. 8 is a diagram showing a state in which an individual is identified from the frame image of FIG. 5. FIG. 9 is a diagram showing a state in which a part is detected from the frame image of FIG. 8. FIG. 10 is a diagram showing a state in which the part of FIG. 9 is extracted. FIG. 11 is a diagram showing a state in which the parts extracted from each of the frame images of FIG. 5 to FIG. 8 are tracked. FIG. 12 is a diagram showing a tracking image in which a marker is added to the frame image at time t3 of FIG. 8.

[0052] In this information processing system, the inside of a cage C is captured by a camera 1, and the captured video is transmitted to an image processing device 2. In the image processing device 2, the image acquisition unit 51 acquires frame images G1, G2, and G3 from the moving image input from the camera 1 in chronological order of times t1, t2, and t3, as shown in FIGS.

[0053] Next, in the part extraction unit 52, the individual recognition unit 71 binarizes the unit images (pixels) included in the frame image G1 at time t1, for example, and partitions the background (cage part) from the white body parts of the mice X1 and X2 to recognize the regions for each individual. In this example, as shown in Fig. 8, a region 81-1 of the body part of the mouse X1 and a region 91-1 of the body part of the mouse X2 are recognized.

[0054] Next, as shown in FIG. 9, one or more parts 82-1, 92-1 of each of the areas 81-1, 91-1 are detected by the part detection unit 72 as tracking points for tracking behavior.

[0055] 9, the center of gravity of region 81-1 is calculated by part detection unit 72, and two-dimensional coordinates (x1, y1) indicating part 82-1 of the center of gravity of region 81-1 on frame image G1 are detected as a tracking point. Part 82-1 is assigned the individual ID "0".

[0056] Similarly, the center of gravity of the region 91-1 is calculated, and the two-dimensional coordinates (x2, y2) indicating the part 92-1 of the center of gravity of the region 91-1 on the frame image G1 are detected as a tracking point. The part 92-1 is assigned the individual ID "1". In this way, one or more parts of the animal are detected, and body movements are converted into data from the parts obtained as a result of the detection.

[0057] Note that the method of determining the center of gravity of the area and using its position as a tracking point as in this embodiment is just one example, and other than this, one or more of the animal's left and right eyes, nose, left and right front paws, left and right hind paws, the joints of each leg, the outline or center of the ears, the spine, the center or hairline or tip of the tail, etc. may be detected and used as tracking points.

[0058] In this way, by detecting one or more parts of an animal, tracking those parts over time, and comparing them with the learned model in model DB42, it is possible to identify an animal with a skeleton that moves in that way, in other words, to identify the type of animal that the animal is.

[0059] The body parts (tracking points) detected in this manner are extracted, and tracking data 101 including the extracted parts 82-1, 92-1 and individual IDs is generated, as shown in FIG.

[0060] The process of generating the above-mentioned tracking data 101 is also performed on the frame images G2 and G3, and by analyzing them chronologically, it is possible to digitize the positional relationship of each part of each individual mouse X1, X2 at any given time point and the frequency with which certain conditions are satisfied.

[0061] For example, as shown in Fig. 11, by superimposing tracking data 101 having parts generated corresponding to frame images G1 to G3, a movement trajectory associated with the animal's movement can be obtained. That is, the movement trajectory of mouse X1 is obtained as vector data 111-1 and 111-2 indicating the movement trajectory of parts 82-1, 82-2, and 82-3. Also, the movement trajectory of mouse X2 is obtained as vector data 112-1 and 112-2 indicating the movement trajectory of parts 92-1, 92-2, and 92-3.

[0062] Then, the marker image generating unit 55 generates an image in which a marker indicating an animal's body part, an individual ID, and the like are superimposed on the unit image, thereby displaying a tracking image G3A as shown in FIG. In this tracking image G3A, for example, a marker (circle mark) indicating the part 82-3 of the mouse X1, the individual ID "0", and the trajectory line 120 are added to the frame image G3 (see FIG. 7) at the time t3. Additionally, in the tracking image G3A, a marker (triangle mark) indicating the part 92-3 of the mouse X2, the individual ID "1", and the trajectory line 121 are displayed.

[0063] By holding the trajectory lines 120, 121 from times t1, t2 of the frame images G1, G2 for a predetermined time (for example, about 0.5 seconds) and playing back the tracking image, it is possible to play back a video in which the mice X1, X2 move while retaining the trajectory lines 120, 121.

[0064] Here, with reference to Figs. 13 to 15, several examples will be described in which the behavior of each individual mouse is determined from the positional relationship of one or more parts of the mouse, and sociality, interactions, and relationships are detected. Fig. 13 is a diagram showing an example of detection of a water drinking behavior of a mouse, Fig. 14 is a diagram showing an example of detection of a food eating behavior of a mouse, and Fig. 15 is a diagram showing an example of detection of mutual interference behavior between mice.

[0065] As shown in Figure 13, the image processing device 2 extracts specific parts of the mouse (in this example, the nose and the left and right eyes) as tracking points from a video captured by a camera 1 of a cage containing a mouse, and tracks the positions of the tracking points (coordinates x, y) over time, thereby obtaining coordinates indicating the positions of the mouse's nose and the left and right eyes for each individual mouse for each frame image.

[0066] In the image processing device 2, the behavior determination unit 56 determines the behavior of the animal based on whether or not a time-series change in the position of a predetermined part of the animal matches a predetermined condition. Specifically, the behavior determination unit 56 determines that the mouse is drinking water when the distance between the position (coordinates) of the mouse's nose and the position (coordinates) of the water spout attached to the cage wall remains zero or close to zero for a certain number of frames, for example, frames No. 150 to 156 (the part in brackets) in Figure 13.

[0067] Furthermore, in frames No. 150 to 156 ({part in brackets) in Figure 14, if the distance (coordinates) between the position (coordinates) of the mouse's nose and the position (coordinates) of the food area located in the cage remains zero or close to zero, the behavior determination unit 56 determines that the mouse is engaged in eating behavior.

[0068] Furthermore, in frames No. 150 to 156 (part in {brackets) in Figure 15, if a close proximity state in which the distance between the positions (coordinates) of the mouse's nose and the positions (coordinates) of the mouse's nose is zero or close to zero continues, the behavior determination unit 56 determines that this is mutual interference behavior between the mice.

[0069] As described above, according to the image processing device 2 in the information processing system of the first embodiment, a part of the body of each of the two mice X1 and X2 is extracted from a moving image capturing the two mice X1 and X2 moving in the cage C, and the positions of the extracted parts are analyzed in chronological order to identify each of the mice X1 and X2. Therefore, each of the mice X1 and X2 can be distinguished from a moving image capturing the mice X1 and X2 moving in a certain behavioral range. In addition, the system determines mouse behavior, such as the mouse's water drinking behavior, food eating behavior, and mutual interference between mice in the same behavioral range, making it possible to consider mouse habits, ecology, and master-slave relationships between mice (dominance, subordination, etc.). The individual identification section 54 can correctly identify individuals from the continuity of the part positions extracted in the previous stage, under the assumption that there is no large difference in part positions between previous and subsequent frames. Since the individual identification unit 54 identifies an individual based on the positional information of the outer contour and parts, the individual can be correctly identified regardless of the brightness or background of the shooting environment.

[0070] Next, a second embodiment will be described with reference to Fig. 16. In describing the second embodiment, the same components as those in the first embodiment shown in Fig. 3 will be denoted by the same reference numerals, and description thereof will be omitted. FIG. 16 is a functional block diagram showing a second embodiment of the information processing system in FIG. 1, that is, a second embodiment of the functional configuration of the image processing device 2 in FIG.

[0071] The image processing device 2 includes a contour detection unit 57. The contour detection unit 57 detects the contours (body outlines, etc.) of the bodies of one or more animals for each of a plurality of unit images, using a skeleton estimation model selected from a plurality of types of skeleton estimation models in the model DB 42. The contour detection unit 57 includes an individual recognition unit 91 and a contour specification unit 92 . The individual recognition unit 91 recognizes, in each of the plurality of unit images, an area including a unit image having a change (movement) as one behavioral individual. Specifically, the individual recognition unit 91 binarizes each unit image and recognizes those with a color different from the background image as part of a behaving individual. Preferably, a part where the color changes when compared with the previous and next frame images in the time series is regarded as the boundary of the region. The contour specifying unit 92 specifies, for each of the plurality of unit images, a region including one recognized as a single behavioral individual as the body contour (body outline or the like) of each of the one or more mice X1, X2. The data frame generating unit 53 generates a data frame (see FIG. 22) indicating the probability that a change in the outer contour accompanying an animal's behavior is a specific behavior.

[0072] The individual identification unit 54 identifies each individual of one or more animals in each of the multiple unit images based on the output obtained by inputting a time series of the body contours of each of the one or more animals detected from each of the multiple unit images by the contour detection unit 57 into the skeletal estimation model of the model DB42. The individual identification unit 54 analyzes one or more contours detected from each of the multiple unit images by the contour detection unit 57 in time series, and identifies each individual of the one or more animals in each of the multiple unit images based on the analysis results. Specifically, the individual identification unit 54 refers to the model DB 42 and analyzes one or more contours detected from each of the multiple frame images for each time series, thereby identifying each individual of the mice X1 and X2, and outputs the individual information of the mice X1 and X2 obtained as a result of the identification (such as an individual ID and the position of the contour) to the behavior determination unit 56. In other words, the individual identification unit 54 identifies which of the two mice X1 and X2 in the cage the detected contour belongs to.

[0073] The learning model of the model DB42 in this second embodiment is the result of machine learning performed on videos and still images of individual mice prepared in advance, and stores learned learning data for identifying a mouse from a newly input video or still image and outputting information about the identified individual (individual ID, outer contour position, etc.).

[0074] Specifically, for example, when a video obtained by photographing each mouse in action is input into model DB42, the individual ID of each of the one or more mice included in the video and the position of the outer periphery (body outline) in the video of the mouse with that individual ID are output. In other words, by using the learning model in the second embodiment, which is a learning model that uses a skeleton estimation technique, it is possible to detect the position of the outline of an individual from the area of ​​each identified individual.

[0075] Based on the individual information input from the individual identification unit 54, the behavior determination unit 56 digitizes, for example, the general behavior of each individual at any given time and the frequency with which certain conditions are met, and compares the individual behavior and conditions with the data in the material DB 43. Specifically, when the probability value of a specific behavior in the data frame satisfies any one of one or more preset conditions, the behavior determination unit 56 determines the behavior according to the satisfied condition. Here, the condition of 1 or more refers to a condition in which, for example, after one mouse performs the general behavior of curling up, the other mouse performs the general behavior of rubbing up against the curled-up mouse, and this occurs a certain number of times or more. The behavior determination unit 56 refers to the material DB 43, and when the behavior of the mouse identified by the individual identification unit 54 satisfies the condition, it determines that the relationship between the mice is good and that they are about to become a pair.

[0076] As described above, according to the functional configuration of the image processing device 2 in the information processing system of the second embodiment, in addition to the same effects as those of the first embodiment, the following effects can be obtained. That is, from a video in which two mice X1 and X2 housed in a cage C and active are captured, the contours of the bodies of the mice X1 and X2 are detected, the contours thus detected are analyzed in chronological order, and each of the mice X1 and X2 is identified. This makes it possible to distinguish between the mice X1 and X2 that are present and active in the same image.

[0077] Second embodiment Next, an image processing operation executed by an information processing device having the functional configuration of the second embodiment in Fig. 16 will be described with reference to Fig. 17. In describing the image processing operation of the second embodiment, the same operations as those of the first embodiment shown in Fig. 4 are assigned the same step numbers as in Fig. 4, and descriptions thereof will be omitted. Fig. 17 is a flowchart for describing an example of the flow of image processing executed by an information processing device having the functional configuration in Fig. 16.

[0078] In the second embodiment, in step S21, a video is acquired from the camera 1, and in step S22, the contour detection unit 57 detects the body contours (body outline, etc.) of one or more animals for each unit image of multiple frame images contained in the acquired video.

[0079] In step S23, the individual identification unit 54 analyzes the contours extracted from each of the plurality of unit images by the contour detection unit 57 for each time series, and identifies each of the one or more individual animals in each of the plurality of unit images based on the analysis results. The processing from step S24 onwards is the same as that in the first embodiment (FIG. 4).

[0080] According to the operation of the image processing device 2 having the functional configuration of the second embodiment, the contours of the bodies of the mice X1 and X2 are detected for each unit image of a plurality of frame images included in a moving image acquired from the camera 1, and the changes in the contours over time in accordance with the behavior of the mice X1 and X2 are analyzed to identify each of the mice X1 and X2, thereby making it possible to distinguish each of the mice X1 and X2 appearing in the same image. In addition, by monitoring the changes in the contours of the identified mice X1 and X2 over time and detecting specific behaviors of the mice X1 and X2 when predetermined conditions are met, it is possible to discover not only existing but also new ecology.

[0081] Next, the procedure of image processing by the information processing device in this information processing system will be described with reference to FIGS. Fig. 18 is a diagram showing an example of a frame image acquired from a video. Fig. 19 is a diagram showing an example of a frame image acquired from a video after the frame image of Fig. 18. Fig. 20 is a diagram showing a tracking image in which an object showing an outline and an individual marker are added to a frame image at time t4 in Fig. 18. Fig. 21 is a diagram showing a tracking image in which an object showing an outline and an individual marker are added to a frame image at time t5 in Fig. 19. Fig. 22 is a diagram showing an example of detection of a scratching behavior. 18 to 21, one of the two mice shown in the frame images will be described as the mouse X1-1 that appears in frame image G4 at time t4 and the mouse X1-2 that appears in frame image G5 at time t5. The same applies to the other mice, X2-1 and X2-2.

[0082] In this information processing system, the inside of a cage C is captured by a camera 1, and the captured video is transmitted to an image processing device 2. In the image processing device 2, the image acquisition unit 51 acquires, in chronological order, a frame image G4 at time t4 shown in FIG. 18 and a frame image G5 at time t5 shown in FIG. 19 from the moving image input from the camera 1.

[0083] Next, in the contour detection unit 57, the individual recognition unit 91 binarizes the unit image included in the frame image G4 at time t4, for example, so that the background (the cage part) and the white body part of the mouse become different colored areas, and the area is recognized for each individual. As a result, the body part area of ​​each of the two mice is recognized individually.

[0084] Figure 20 shows an example of the display of an individually identified tracking image G4A. In this example of the display of tracking image G4A, a blue object 121-1 is displayed on the outline (body outline) of mouse X1-1, and a red object 122-1 is displayed on the outline (body outline) of mouse X2-1. Also displayed in the tracking image G4A is "mouse0.996" as an individual ID for identifying the individual mouse X1-1, along with a marker such as a frame 131-1 surrounding the individual. Furthermore, in the tracking image G4A, "mouse0.998" is displayed as an individual ID for identifying the individual mouse X2-1, along with a marker such as a frame 132-1 surrounding the individual.

[0085] Similarly, in frame image G5 at time t5, the unit image (pixel) is binarized, the background (cage part) and the white body part of the mouse are partitioned, and the area is recognized for each individual mouse.

[0086] In the example of individually identified tracking image G5A in FIG. 21, a blue object 121-2 is displayed on the contour (body outline) of mouse X1-2, and a red object 122-2 is displayed on the contour (body outline) of mouse X2-2. Also displayed in the tracking image G5A is "mouse0.996" as an individual ID for identifying the individual mouse X1-2, along with a marker such as a frame 131-2 surrounding the individual. Furthermore, in the tracking image G5A, "mouse0.998" is displayed as an individual ID for identifying the individual mouse X2-2, along with a marker such as a frame 132-2 surrounding the individual.

[0087] In this way, the outlines of the body parts of two mice are detected, and the individual mice are identified from the changes in those outlines over time. Then, when each individual mouse fulfills a predetermined condition at any given time, a specific behavior of the mouse is determined, and the reason for each mouse's behavior can be determined.

[0088] Here, an example of determining the behavior of each individual mouse from the positional relationship of the mouse's outline and detecting sociality, interaction, and relationship will be described with reference to FIG. FIG. 22 shows an example of detecting a scratching behavior. Frame images are acquired in order starting from frame No. 1 in Figure 22, and after the mouse has been individually identified, behavior determination unit 56 calculates the probability for each frame image as to whether or not the mouse is performing a specific behavior, and converts the resulting probability into a data frame for each frame image. Then, based on the probability of the data frame, the mouse behavior is labeled (classified), and the identification information is assigned to the data frame and stored in the image DB 41.

[0089] For example, in the data frames of frames No. 4502 to 4507 in FIG. 22, when the calculated probability exceeds a certain threshold, the behavior determination unit 56 determines that the mouse has performed a specific behavior, and assigns identification information indicating the specific behavior (e.g., predict “1”) to the data frames of frames No. 4502 to 4507. If the time series changes in the mouse's outer shell (mouse behavior) match the conditions previously set in the material DB 43, the behavior is determined to be a specific mouse behavior derived from the conditions, such as the mouse scratching the cage floor, i.e., scratching behavior.

[0090] According to the image processing device 2 of the second embodiment, the probability of whether or not an individually identified mouse is performing a specific behavior is calculated for each frame image, and based on the probability obtained as a result of the calculation, the data frame at the time when the mouse performed the specific behavior is labeled, making it possible to detect not only the mouse's expected behavior but also unexpected behavior. As a result, it is possible to discover general and new behaviors of multiple mice housed in a cage while distinguishing and observing each of them.

[0091] Third embodiment Next, a third embodiment will be described with reference to FIGS. In the above first and second embodiments, a model was constructed only for a specific animal species, behavior (white mice, scratching behavior, etc.), and shooting environment, output as RAW data (positional information of each body part, etc.), and a condition judgment program was constructed. However, this third embodiment expands the application to multiple animal species, behaviors, and shooting environments, improving practicality. In the third embodiment, annotation for acquiring RAW data (creation of training data for determining the position of the acquisition site), construction of judgment conditions (mathematical models) for each behavior using the RAW data, programming and modularization for application development, and extension of the program for detecting the movement, eating, and drinking of the white mouse were performed.

[0092] In addition, while the above first and second embodiments realize the minimum required tasks using a proprietary hardware and software environment, in the third embodiment, a user interface environment (hereinafter referred to as a "UI environment") that is easy to operate and has consistency in business processes is constructed to increase versatility and allow general users with low IT literacy to use it stably. Additionally, the system was built as a cloud service, that is, a server-client system, with the expectation that the system's execution environment would operate stably regardless of the user's (client's) conditions, and that specifications and costs would be optimized according to the frequency and load of tasks being executed.

[0093] First, an overview of an information processing system according to the third embodiment will be described with reference to FIG. FIG. 23 is a diagram showing a third embodiment of an information processing system, and is a diagram showing an outline of a business model in which the information processing systems of the first and second embodiments are expanded for commercial use.

[0094] As shown in FIG. 23, the information processing system of the third embodiment is configured so that a device on the side of a client 300 of a requester Y communicates with a server 200 of a contractor who undertakes the request. Upon receiving a request from a client Y, the server 200 analyzes the video uploaded by the client Y and transmits the analysis result to the client Y.

[0095] The devices on the client 300 side include a camera 310 and a cage 311 that constitute the animal's photography environment, and a client computer 320 (hereinafter referred to as "PC 320") that collects video images captured by the camera 310 and that are the subject of analysis. Cage 311 is a container for allowing an animal to move around within a certain range. Camera 310 captures an image of the animal moving around within cage 311. The environment in which the animal is photographed can be changed as appropriate depending on the request.

[0096] The PC 320 captures video (image) data captured by the camera 310 and stores it locally (internal storage, etc.). The PC 320 uploads the video data 321 stored locally to the server 200 and requests an analysis of the behavior of animals included in the video. In response to the request, the PC 320 obtains from the server 200 information on the analysis results (CSV file 330) and moving image data (processed moving image 331) that has been processed by adding markers M to the positions of the animal's eyes. The PC 320 creates materials (graphs 322) for analyzing the data in the CSV file 330, and includes the processed video 331 as part of the analysis materials.

[0097] The server 200 includes an image analysis unit 450, a web service unit 451, a storage 410, and a data warehouse 411 (hereinafter referred to as "DWH 411"). The image analysis unit 450 includes the functional configuration shown in FIG. 3 in the first embodiment and the functional configuration shown in FIG. 16 in the second embodiment, and performs image analysis processing.

[0098] The web service unit 451 has a function of authenticating login information input from the PC 320, a function of searching for an analysis target, a function of outputting the search results, a function of adding analysis data, and a function of displaying the analysis results. A specific functional configuration will be described later. In addition, the web service unit 451 has a function for performing encrypted communication, access source IP communication, leased line communication, etc., thereby ensuring security. That is, the web service unit 451 realizes an interface with the PC 320 .

[0099] The storage 410 stores video data requested for analysis via the web service unit 451 and video data currently being analyzed. The DWH 411 stores various related data and systematically archives the results of processing in conjunction with the data in the storage 410. This makes it possible to reuse data and reduce the number of animal experiments.

[0100] Here, a data processing procedure (flow) of the information processing system according to the third embodiment will be described. In step S101, video of the animal's activities in the cage 311 is captured and the video data 321 is stored locally in the PC 320. In step S102, when requesting the contractor to analyze the video, the video data 321 stored in the PC 320 is uploaded to the server 200.

[0101] In the server 200 , the web service section 451 stores the video data 321 uploaded from the PC 320 in the storage 410 . In steps S101 and S102, the captured video data 321 is temporarily stored in PC 320 and then uploaded to server 200. However, in the case of a video that is captured continuously for 24 hours, the video data 321 captured by camera 310 may be uploaded directly to server 200, as in step S103.

[0102] In step S104, in the server 200, the image analysis unit 450 reads out the video data stored in the storage 410, performs the requested analysis of the video data 321, and stores in the storage 410 the processed video 331 generated by processing the video during the analysis and the CSV file 330 of the analysis results, and outputs an analysis completion notification to the PC 320. On receiving the analysis completion notification, PC 320 displays a search screen. When the video data to be analyzed is specified, in step S105, web service unit 451 searches for the specified video data and downloads processed video 331 and the CSV file 330 of the analysis results to PC 320.

[0103] In step S106, the PC 320 creates a graph 322 for analysis using the processed moving image 331 downloaded from the server 200 and the CSV file 330 of the analysis results, and attaches the processed moving image 331 to the graph 322 as supporting material.

[0104] In the above steps S105 and S106, the CSV file 330 (numerical data) of the analysis results is downloaded to the PC 320, and the graph 322 is created from the CSV file 330. Alternatively, for example, as in step S107, depending on the contents of the request, the graph 322 can be created from the CSV file 330 in the server 200, and the graph 322 can be downloaded to the PC 320 as the analysis result.

[0105] Next, a functional configuration of the information processing system of the third embodiment in Fig. 23 will be described with reference to Fig. 24. Fig. 24 is a functional block diagram showing the functional configuration of the information processing system of the third embodiment in Fig. 23. In addition, when explaining the functional configuration of the information processing system of the third embodiment, the same components as those in the functional configuration of the first embodiment shown in FIG. 3 and the functional configuration of the second embodiment shown in FIG. 16 are denoted by the same reference symbols, and their explanations are omitted.

[0106] 24, an authentication DB 44 is stored in one area of ​​the storage unit 18 of the server 200. The authentication DB 44 stores authentication information for the requester Y to log in to the server 200. The authentication information is a login ID and a password as identification information for identifying the requester Y, etc. Furthermore, the model DB 42 stores in advance a plurality of types of learning models that are generated or updated by machine learning or the like. The learning models are, for example, a skeleton estimation model, an individual identification model, and a mathematical model. The plurality of types of learning models are provided, for example, for each type of animal to be analyzed, for each image capturing direction, and for each color of the animal.

[0107] When the processes corresponding to steps S101 to S107 in FIG. 23 are executed, in the CPU 11 of the server 200, an image analysis unit 450, a web service unit 451, and a process unit 452 function. The image analysis unit 450 performs image analysis processing using a model selected for the designated image data from the video data 321 acquired at the request of the client Y. The image analysis unit 450 reads out the image to be analyzed, specified by the specification unit 471, from the image DB 41, and performs analysis processing of the image using a learning model (skeleton estimation model, individual identification model, mathematical model, etc.) selected from multiple types of learning models by the model selection unit 472 and extracted from the model DB 42. The image processing executed by the image analysis unit 450 is processing such as individual identification, contour identification, and behavior analysis of animals in a moving image, as shown in the first and second embodiments. The web service unit 451 performs login authentication when the client Y logs in to the server 200, searches for the results of the video analysis requested by the client Y, outputs the search results, adds the analysis data, and controls the display of the analysis results.

[0108] The web service unit 451 includes an authentication unit 461, a search unit 462, a search result output unit 463, an analysis data addition unit 464, an analysis result display control unit 465, and the like.

[0109] The authentication unit 461 compares the input login information with the authentication information in the authentication DB to authenticate the user. If the information matches as a result of comparing the two pieces of information, the authentication unit 461 permits the client Y to log in to the server 200.

[0110] The search unit 462 displays the search screen 251 (see FIG. 25) on the PC 320 of the authenticated client Y, searches the image DB 41 for analysis results in response to a search request from the search screen 251, and passes the search results to a search result output unit 463. On the search screen 251, it is possible to display a list of search results, add analysis data, display the analysis results, and the like.

[0111] A search result output unit 463 outputs the search results obtained by the search unit 462 to the PC 320 and displays them as a list on the search screen 251 of the PC 320 .

[0112] The analysis data adding unit 464 displays an analysis data adding screen 261 (see FIG. 26) on the PC 320 when an analysis data adding operation is performed on the search screen 251. By performing the analysis data addition operation on the analysis data addition screen 261, the moving image data to be analyzed is uploaded to the image processing device 2. In the image processing device 2, the uploaded moving image data to be analyzed is additionally registered in the image DB 41. After uploading, the analysis conditions for the new file added to the image DB 41 can be specified on the analysis data addition screen 261.

[0113] The analysis data adding unit 464 includes a designation unit 471 and a model selection unit 472 . The designation unit 471 designates analysis attributes for the image to be analyzed (type of analysis object (e.g., mouse, rat, etc.), imaging direction of the analysis object (e.g., upward, oblique upward, horizontal, etc.), color of the analysis object (e.g., white, black, etc.)).

[0114] Specifically, the designation unit 471 displays the analysis data addition screen 261 of FIG. A file column and a setting column are provided on the analysis data addition screen 261. The file column provides an icon of the file of the image to be analyzed and a bar graph indicating the progress (progress) of the analysis. The setting column provides radio buttons (designation buttons) for the client Y to specify analysis attributes such as the type of animal contained in the image to be analyzed (e.g., mouse, rat, etc.), the imaging direction of the animal to be analyzed (e.g., above, diagonally upward, sideways, etc.), and the color of the animal to be analyzed (e.g., white, black, etc.).

[0115] When the requester Y uploads an image to be analyzed to the image processing device 2, an icon of the uploaded image file to be added is displayed in the file field of the analysis data addition screen 261.

[0116] Then, by the client Y configuring the image settings (such as the type of animal contained in the image to be analyzed, the imaging direction of the animal to be analyzed, and the color of the animal to be analyzed) using the radio buttons provided in the corresponding settings column below the image file icon, the accuracy of the image analysis can be improved, enabling accurate identification of the type and individual animals contained in the image, behavioral analysis, etc.

[0117] The model selection unit 472 selects a model to be applied to the part extraction unit 52 in FIG. 3 or the outline detection unit 57 in FIG. 16 from among the multiple types of skeleton estimation models stored in the model DB, based on the analysis attributes of the image specified by the specification unit 471, and also selects a model to be applied to the individual identification unit 54 from among the multiple types of individual identification models stored in the model DB. The analysis result display control unit 465 displays the CSV file 330 of the analysis results downloaded from the server 200, the processed moving image 331, and the like.

[0118] The process unit 452 manages image data for analysis and unprocessed data. The process unit 452 includes an upload data management unit 491 and an unprocessed data management unit 492 . The upload data management unit 491 moves the image data uploaded from the PC 320 to the image DB 41 and updates the management file. In addition, the upload data management unit 491 has a function of starting every minute by cron, a function of preventing duplicate start, and the like.

[0119] The unprocessed data management unit 492 checks the unprocessed data in the management file. The unprocessed data management unit 492 copies the unprocessed data. The unprocessed data management unit 492 performs AI processing in the processing directory. The unprocessed data management unit 492 stores the result file in the corresponding directory. The unprocessed data management unit 492 creates a CSV file 330 of the analysis result and stores it in the corresponding directory. In addition, like the uploaded data management unit 491, the unprocessed data management unit 492 has a function of starting every minute by cron, a function of preventing duplicate start-up, a function of updating the management file, etc.

[0120] When the processes corresponding to steps S101 to S107 in FIG. 23 are executed, in the CPU 141 of the PC 320, a login management unit 421, a screen control unit 422, an analysis request unit 423, and an analysis result display control unit 424 function. The login management unit 421 displays a login screen on the PC 320, and transmits the login information entered by the client Y on the login screen to the server 200 to request login authentication. The screen control unit 422 displays a search screen on the PC 320, transmits the search keywords entered on the search screen by the requester Y to the server 200 to request a search, and displays the search results for the request on the search screen. A list of search results is displayed on the search screen. In addition, the search screen allows the addition of analysis data, display of analysis results, etc. The analysis request unit 423 displays an analysis data addition screen on the PC 320, and transmits the files and analysis conditions designated by the requester Y on the analysis data addition screen to the server 200 to request an analysis. The analysis result display control unit 424 displays an analysis result display screen on the PC 320, and displays the CSV file 330 of the analysis results downloaded from the server 200 and the processed moving image 331 on the analysis result display screen.

[0121] Hereinafter, the operation of the information processing system of the third embodiment will be described with reference to FIGS. Fig. 25 is a diagram showing a search screen displayed on a PC. Fig. 26 is a diagram showing an analysis data addition screen displayed as a pop-up on the search screen. Fig. 27 is a diagram showing a video confirmation screen displayed as a pop-up on the search screen.

[0122] The search screen shown in Fig. 25 is displayed on the PC 320 (see Fig. 24). The search screen 251 is provided with input fields for inputting search conditions such as keywords and time period, an analysis data list, an analysis data addition button, a video confirmation button, a CSV output button, and the like. The analysis data list lists the analysis data with each item such as the file name, registrant, date and time, and analysis of the analysis data. The analysis item indicates the current analysis state of the analysis data, for example, "Analyzed" or "In Progress". Selection buttons for selecting each analysis data are provided on the left side of the analysis data list, and by operating the button for the selected analysis data, it is possible to add analysis data, check the video, output in CSV format, etc.

[0123] On this search screen 251, when a large amount of analysis data is registered in the server 200, the analysis data can be narrowed down using search conditions, and analysis data displayed in the analysis data list can be selected (specified) using the selection button. Then, by pressing either the Add Analysis Data button, the Video Confirmation button, or the CSV Output button, the user can perform the process corresponding to the button operation (add analysis data, confirm video, output CSV, etc.).

[0124] (Added analysis data) On the search screen 251 in FIG. 25, when desired analysis data is selected (specified) with the selection button and then, for example, an analysis data addition button is pressed, an analysis data addition screen 261 shown in FIG.

[0125] The analysis data addition screen 261 is provided with the file of analysis data to be added, an analysis attribute setting field for setting (specifying) the type of video contained in each file, a back button, a cancel button, a registration button, etc. The analysis attribute setting field includes buttons for selecting, for example, the type of animal, the direction in which the animal is photographed, the color of the animal, etc., and these can be set (specified) using the respective buttons. On this analysis data addition screen 261, by selecting and setting the file to be analyzed for video data uploaded as new analysis data, subsequent analysis of the analysis data can be automatically executed. At this time, based on the type of animal, the imaging direction of the animal, the color of the animal, etc., set as analysis attributes on the analysis data addition screen 261, a learning model to be used for the animal to be analyzed is selected from multiple types of learning models (skeletal estimation model, individual identification model, and mathematical model) stored in the model DB 42.

[0126] (Check the video) On search screen 251 in FIG. 25, when desired analysis data is selected (specified) with the selection button and then, for example, the video confirmation button is pressed, a video confirmation screen 271 shown in FIG. The video confirmation screen 271 is provided with a playback area for playing back the analysis data to be added, a back button, a download button, etc. In the playback area, an image (still image) of the selected analysis data (video data) stopped and a play button (triangle icon) are provided, and by clicking the play button (triangle icon), the still image starts moving and is played as a video. This video confirmation screen 271 allows you to play and check video data uploaded as new analysis data when it is difficult to determine from the file name alone whether it is data to be analyzed or an edited video that has already been analyzed. In addition, by pressing the download button, the analysis data displayed in the playback area can be downloaded. This download function is effective when the processed video 331 is to be used on the PC 320 side.

[0127] (Advantages of using this information processing system) The advantages of utilizing this information processing system will be described with reference to FIG. FIG. 28 is a diagram showing an example of a report created from a video. This information processing system makes it possible to output quantitative data on the opening and closing of a mouse's eyes, which is minute and cannot be confirmed by visual inspection, enabling the confirmation of information with unprecedented precision and understanding that was previously impossible. For example, as shown in FIG. 28, when video data 281 of a mouse is uploaded to server 200 from client Y's PC 320 and a request is made to analyze the state of the mouse's eyes, markers M are set at the positions of the mouse's eyes by the part detection unit 72, and it is identified that the area of ​​the eye becomes larger when the mouse's eyes are opened and that the area becomes smaller when the mouse's eyes are closed, in other words, by identifying whether the eyes are open or closed, a graph 283 showing, for example, the change in the area of ​​the eye with time can be created as report 282 and provided to client Y. In the report 282, for example, it is possible to output the number of times the eyes were closed per certain period of time, the duration of eye closure (how many times the eyes were closed per minute), and other data according to the request. That is, the information processing system can quantitatively capture very slight changes in a region, can automatically process long videos, and can improve throughput.

[0128] (Robust security) The security of this information processing system will be described with reference to FIG. FIG. 29 is a diagram showing an example of a closed network connection configuration that provides robust security. This information processing system can be configured with a security level selected according to the user's policy and cost. Note that communication encryption is required, and https is a prerequisite. Examples of system configurations according to security level include "Plum," "Take," and "Matsu." "Plum" allows access by restricting access only through the in-house proxy GIP. "Take" allows access via a reverse proxy using a client certificate. "Matsu" connects to the in-house environment via a closed network via DX (direct connect). An example of the configuration of the above-mentioned "Matsu" closed network connection is shown in FIG.

[0129] (Development of this information processing system) The details of this information processing system will be described with reference to FIG. FIG. 30 is a diagram showing a dashboard screen of the information processing system. As shown in FIG. 30, the present information processing system has a function of displaying a dashboard screen 500 that allows easy information management. The dashboard screen 500 has features such as the ability to be expanded according to the animal species and experimental environment, the ability to output in formats other than experimental reports, and the ability to use artificial intelligence to discover and notify users of features that would go unnoticed by humans.

[0130] Specifically, on the dashboard screen 500, various documents necessary for carrying out an experiment can be created, such as an application for approval to install a feed storage facility, an animal experiment plan, an animal experiment result report, self-inspection and evaluation items, and the like. In terms of tailoring it to the business, it is possible to expand the range of animals that can be used and the content of experiments, and it is also possible to output in formats other than experiment reports.

[0131] Furthermore, the dashboard screen 500 allows for the expansion of "discoveries" through the deepening of machine learning. For example, "discoveries" within the range of human cognition and "discoveries" beyond the range of human cognition are possible. As for machine learning and function deepening, it is possible to expand the applicable animals and experimental contents. Artificial intelligence can be used to discover and notify features that humans would not notice.

[0132] In summary, in the information processing system of the third embodiment, when analysis attributes for an image to be analyzed (type of analysis object (e.g., mouse, rat, etc.), imaging direction of the analysis object (e.g., upward, diagonal upward, sideways, etc.), color of the analysis object (e.g., white, black, etc.)) are specified, the model selection unit 472 selects a skeletal estimation model to be applied to the part extraction unit 52 of FIG. 3 or the outline detection unit 57 of FIG. 16 from among multiple types of skeletal estimation models in the model DB 42 based on the analysis attributes of the specified image to be analyzed, and also selects an individual identification model to be applied to the individual identification unit 54 of FIG. 3 or FIG. 16 from among multiple types of individual identification models, and analyzes the image using the selected skeletal estimation model and individual identification model in response to an image analysis instruction, so that the type of animal can be correctly identified using a model appropriate for the analysis object, and analysis results such as the animal's behavior can be obtained. In addition, by being applicable to multiple animal species and experimental contents, and providing functions such as management and downloading of analysis data in server 200, it becomes possible to reuse the analysis data, thereby significantly reducing the number of individual animals actually used in experiments. In addition, the following effects can be achieved: For example, analysis data is automatically labeled (tagged) based on specified analysis attributes, etc., so when reusing the analysis results, the analysis data can be searched for using the tags, making it easier to search for the analysis data. If only a single model is used for multiple analysis attributes, etc., it is necessary to re-learn the existing model to update it and re-examine its accuracy. However, by maintaining multiple models for multiple analysis attributes, etc., a new model can be created when a new attribute appears, making model learning compact.

[0133] The above-described series of processes can be executed by hardware or software. In other words, the functional configurations in FIG. 3, FIG. 16, and FIG. 24 are merely examples and are not particularly limited. That is, it is sufficient that the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and the functional blocks and databases used to realize these functions are not limited to the examples in Figures 3, 16, and 24. The locations of the functional blocks and databases are also not limited to those in Figures 3, 16, and 24, and may be arbitrary. The functional blocks and databases of the image processing device 2 and server 200 may be transferred to the camera 1, PC 320, etc. Furthermore, the image processing device 2, camera 1, and PC 320 may be the same hardware.

[0134] Furthermore, for example, when a series of processes is executed by software, a program constituting the software is installed into a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware, or may be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0135] Furthermore, for example, a recording medium containing such a program may not only be composed of a removable medium (not shown) that is distributed separately from the device body in order to provide the program to the user, but may also be composed of a recording medium that is provided to the user in a state in which it is pre-installed in the device body.

[0136] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically according to the order, but also processes that are not necessarily performed chronologically but are executed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is composed of a plurality of devices, a plurality of means, etc.

[0137] In the above embodiment, the markers are triangular or circular, but the markers are not limited to these examples and may be of other shapes. The same applies to objects that represent the contours of the body. In the above embodiment, it has been described that the behavior determination unit 56 determines whether or not the conditions defining the positional relationship between the body part of the animal and other specific body parts (watering area, feeding area, body parts of other animals) are satisfied, but the conditions defining the positional relationship between the body part of the animal and other specific body parts are stored in the material DB 43 in the storage unit 18, and the behavior determination unit 56 reads out the above conditions from the material DB 43. Alternatively, a behavior database may be provided separately from the material DB 43, and the above conditions may be stored in the behavior database. In the above embodiment, a machine learning technique is used, but other rules may be set, such as determining that the part farthest from the center of gravity is the nose. In the above embodiment, the center of gravity was detected (extracted) as a body part, but it is also possible to detect (extract) other parts, such as the eyes, nose, ears, feet, or specific parts of bones or joints (eyes, nose, ears, etc.). In the above embodiment, mice have been described as an example of the subject of individual identification, but by expanding the data in the model DB 42 and the material DB 43, it is possible to analyze rats, hamsters, guinea pigs, rabbits, etc. In addition, it is possible to target various animals, such as livestock such as pigs, cows, sheep, and chickens, as well as dogs, cats, monkeys, and humans. In the above embodiment, a part of the body of each of one or more animals is extracted for each of the multiple unit images, the extracted parts are analyzed in time series, and each individual of one or more animals in each of the multiple unit images is identified based on the analysis results, but this order is one example, and other methods may be used, for example, after individual recognition (identification), individual parts may be extracted. Also, individual identification and part extraction may be performed simultaneously. When extracting individual parts after individual recognition (identification), first determine which part of the image each part exists in, crop, and then extract the individual parts. Furthermore, when identifying an individual after detecting parts of multiple individuals, the locations of multiple specific parts within the image are detected, and then a decision is made as to which set each part belongs (in this case, the set of parts of a specific individual) based on an inference of "how those positions change over time." In other words, it is sufficient if the system includes the steps of determining the location of an animal's body parts, identifying the individual, and, as a result, determining the sociality of the individual.

[0138] Below, with reference to Figures 31 to 54, we will explain data extracted from a video (multiple unit images arranged in order on the time axis) that is correlated with various indices (movement trajectory, distribution of locations in each area, time spent in each area, movement distance, total movement amount, body orientation, speed, angular velocity, etc.) to reveal the positions of certain parts of the mouse's (animal's) body (eyes, nose, etc.), the center of gravity, the movement range as the contour (outline) changes over time, the amount of movement, behavioral habits, rhythm, etc. The graphs described below show a pair of graphs, one for the initial period from the start of measurement up to 1320 frames (44 seconds) and the other for the period from the start of measurement up to 18030 frames (approximately 10 minutes).

[0139] First, the relationship between the position of the mouse in the cage and its movement trajectory will be described with reference to Figs. Figure 31 is a graph showing the position of the mouse in the cage and its movement trajectory from the start of measurement to the initial 1320 frames (44 seconds). Figure 32 is a graph showing the same graph as in Figure 31, plotted continuously until the 18030 frames (about 10 minutes).

[0140] 31 and 32, the vertical axis indicates the length of the cage floor in the depth direction, and the vertical axis indicates the length of the cage floor in the width direction, expressed in the number of pixels (PIXCEL). In the graph of the initial movement of 1320 frames in Figure 31, the mouse is avoiding the center of the cage and moving around the edge, suggesting that it is feeling anxious. In the graph of Figure 32, from the start of measurement until 10 minutes later, it can be seen that, as expected, more mice move around the edge of the cage than around the center, but there are also movement lines that show linear movement and areas where the movement lines seem to be tangled. In areas where the movement lines are tangled, it can be seen that the mice remain in that position and are engaged in some kind of behavior (grooming, etc.). By using AI to learn this information, it is possible to identify mouse behavior and mental illnesses, etc.

[0141] Next, the distribution of mice in each of the nine areas into which the cage floor was divided will be described with reference to Figs. The vertical and horizontal axes in Figures 33 and 34 indicate pixels (PIXCEL). Figure 33 is a graph showing the distribution of mouse positions in each of nine areas (numbered 0 to 8) on the cage floor, from the start of measurement until the initial 1320 frames (44 seconds). Figure 34 is a graph showing the graph in Figure 33, which was then drawn continuously until the 18030 frames (approximately 10 minutes).

[0142] Figures 33 and 34 are graphs in which the position of the mouse inside the gauge is plotted every second. The graphs in Figures 33 and 34 are created by managing the coordinate information of the position where the mouse was located on the floor of the gauge, which is divided into nine areas, by assigning identification information of the area containing those coordinates. In the graph of the initial 1320 frames in FIG. 33, there are many plots in the areas near the edges of the gauge (numbers 0, 5, 6, 8, etc.), and it can be seen that the mouse spends most of its time near the edges. In the graph in Figure 34, which was drawn continuously for about 10 minutes, the mouse still mostly stayed near the edge, but the number of plots in area 4 in the center increased, and it can be seen that the mouse was moving to the center of the gauge more frequently. This means that the mouse was becoming more calm than at the beginning.

[0143] Next, the presence time of the mouse in each of the above-mentioned areas (0 to 8) will be described with reference to FIG. 35 and FIG. Figure 35 is a bar graph showing the time the mouse spent in each of the nine areas on the cage floor, from the start of measurement until the initial 1320 frame (44 seconds). Figure 36 is a bar graph showing the same graph as Figure 35, plotted continuously until the 18030 frame (about 10 minutes).

[0144] 35 and 36, the vertical axis indicates the number of plots (COUNTS), and the horizontal axis indicates the region (REGION). The numbers on the horizontal axis correspond to the numbers (0 to 8) of each area in FIG. Figures 35 and 36 are graphs that count the number of plots where mice were present for each area, that is, the number of mice present for each area. When one plot (point) is added in Figures 33 and 34, the bar graph of the area with the corresponding number in Figures 35 and 36 becomes one count higher. From these Figures 33 to 36, it is possible to visually (visibly) see in which areas of the gauge the mouse is staying most (or least) of the time.

[0145] Next, the time spent by the mouse near the edge and near the center of the cage will be described with reference to Figs. 37 and 38. Figure 37 is a graph showing the time the mouse was present in the cage, divided into near the center (area 4 shown in Figure 33) and near the edge (area other than area 4) from the start of measurement until the initial 1320 frame (44 seconds). Figure 38 is a graph showing the graph in Figure 37 plotted continuously up to 18030 frames (approximately 10 minutes).

[0146] The vertical axis of Figures 37 and 38 indicates the number of plots (COUNTS), and the horizontal axis indicates the region (REGION). The region (REGION) is divided into two parts: the center area (area 4) and the edge area (areas other than area 4). In the graph of the initial movement in Figure 37, we can see that the mouse is often near the edge (areas other than area 4). In the graph in Figure 38, although the mouse is overwhelmingly more likely to be near the edge (areas other than area 4), we can see that the mouse is more frequently near the center (areas other than area 4) than at the initial movement.

[0147] Next, the moving distance of the mouse per second (30 frames) will be described with reference to FIGS. Figure 39 is a graph showing the relationship between the distance the mouse moved per second (30 frames) and the period of 1320 frames from the start of measurement. Figure 40 is a graph showing the same graph as Figure 39, plotted continuously for 18030 frames (about 10 minutes).

[0148] In Figures 39 and 40, the vertical axis indicates distance (PICXEL) and the horizontal axis indicates time (FRAME). The graphs in Figures 39 and 40 show the distance (indicated as a picxel on the image) between the location the mouse has passed and the current location every 30 frames, i.e., every second. In other words, the graphs in Figures 39 and 40 show the distance the mouse moved per second; the higher the vertical swing, the greater the distance moved in a short period of time, and the closer the value on the vertical axis is to 0, the longer the mouse remains in place. In addition, in each graph of Figure 39 and Figure 40, the broken line presented near the center of each fluctuation range indicates the moving average. In this way, by taking the average value over a certain range of time, you can find out the mouse movement speed.

[0149] These graphs are time series graphs, so as shown in the graph in Figure 40, when the mouse settles down in the latter half, it stops moving. A mouse that continues to move at a constant speed can be said to be calm. On the other hand, a mouse that suddenly starts running or stops for an abnormally long time can be hypothesized to be in an abnormal mental state.

[0150] Next, the orientation of the mouse's body in the cage will be described with reference to Figures 41 and 42. Figure 41 is a graph showing the relationship between the time from the start of measurement to the initial 1320 frames and the moment-to-moment body orientation of the mouse in the cage. Figure 42 is a graph showing the same graph as Figure 41, plotted continuously up to 18030 frames (about 10 minutes).

[0151] The vertical axis in FIG. 39 and FIG. 40 indicates angle (DEGREE), and the horizontal axis indicates time (FRAME). In the case of the graphs in Figures 39 and 40, if the mouse's body direction at any given time is set to be straight (0 degrees on the vertical axis of the graph), then if the mouse turns to the right from that point on, for example, the graphs in Figures 41 and 42 will swing in the negative direction. For example, if the mouse turns back, the body direction will change by more than 150 degrees, and the graphs will swing wildly.

[0152] Next, the total moving distance (total amount of movement) of the mouse in the cage will be described with reference to Figs. Figure 43 is a graph showing the relationship between the time from the start of measurement to the initial 1320 FRAME (44 seconds) and the total distance moved by the mouse. Figure 44 is a graph showing the same graph as Figure 43, but continued to be drawn up to 18030 FRAME (about 10 minutes).

[0153] 43 and 44, the vertical axis indicates distance (PICXEL) and the horizontal axis indicates time (FRAME). The graph in Figure 43 shows the total distance (total distance traveled), which is a graph that accumulates the distance traveled described above. The start of measurement is set to 0, and the distance traveled by the mouse is added up, ultimately showing the distance traveled from the start of measurement. In the graphs of Figures 43 and 44, while it was difficult to understand the state at each moment in the graph of the moving speed mentioned above (graph of Figure 39), by comparing this graph of total moving distance with the graph of moving speed, it becomes possible to understand, for example, whether the object is stopped at that moment. This graph shows that the final distance traveled can be long or short, but it also shows where along the way the amount of travel suddenly increases. These graphs therefore give you an idea of ​​mouse behavior in terms of both speed and amount of movement. For example, in the graph in Figure 37, the first red dot (the round dot in the figure) fluctuates up and down over time, but when the red dot moves to the side, it indicates that the mouse is not moving, and the value on the speed graph in Figure 37 is close to 0.

[0154] Next, the behavior of a mouse rotating in a cage will be described with reference to Figs. Figure 45 is a graph showing the relationship between the time from the start of measurement to the initial 1320 frames (44 seconds) and the rotational behavior (angular velocity) of the mouse. Figure 46 is a graph showing the same graph as Figure 45, plotted continuously until 18030 frames (about 10 minutes).

[0155] 45 and 46, the vertical axis represents distance x angle (PICXEL x RAD), and the horizontal axis represents time (FRAME). The graphs in Figures 45 and 46 are graphs showing the change in body direction and the angle at which the body direction changed over time, and are calculated by multiplying the distance traveled by the angle at which the body direction changed, for example, 20 degrees, 30 degrees, etc. In the graphs of Figures 45 and 46, if the person is simply stopping in place and spinning around, the value will be 0, but if the person is changing direction and moving further, the graph will fluctuate greatly up and down. From the graphs in Figures 45 and 46, for example, if the body direction is just changing, the behavior is recognized as looking around, but if the mouse stops in place and turns its body direction, it shows a mental state where it is desperately trying to confirm the surrounding situation. By not only comparing the mouse's behavior with the body direction just before, but also taking into account the movement, a different behavior can be seen.

[0156] Next, the speed at which the mouse moves within the cage will be described with reference to Figures 47 and 48. Figure 47 is a graph showing the relationship between the time from the start of measurement to the initial 1320 frames (44 seconds) and the speed of mouse movement. Figure 48 is a graph showing the same graph as Figure 47 when it is continued to be drawn up to 18030 frames (about 10 minutes).

[0157] The vertical axis in Figures 47 and 48 indicates distance / time (PICXEL / SEC), and the horizontal axis indicates time (FRAME). The graphs in Figures 47 and 48 are conceptually similar to the movement speed graphs in Figures 39 and 40. The graphs in Figures 39 and 40 measure the distance every 30 frames. For example, although data exists between 0 frame, 30 frame, and 60 frame, the graphs in Figures 39 and 40 only compare frames 0 and 30, and frames 30 and 60 to calculate the movement speed, whereas the graphs in Figures 47 and 48 calculate the speed by taking the difference between frames 0 and 1 and dividing it by time, which can be said to be a more detailed speed indicator. Mice and humans experience time at different speeds, with one human second equivalent to a few tenths of a second in a mouse. The graphs in Figures 39 and 40 output speed per second, but this is based on human sensations, so the graphs in Figures 47 and 48 output speed based on a more detailed mouse sensation, which allows for a better representation of mouse behavior.

[0158] Next, the angular velocity at which the mouse rotates within the cage will be described with reference to Figures 49 and 50. Figure 49 is a graph showing the relationship between the time from the start of measurement to the initial 1320 frames (44 seconds) and the angular velocity of the mouse movement. Figure 50 is a graph showing the same graph as Figure 49, but continued to be drawn up to 18030 frames (about 10 minutes).

[0159] 49 and 50, the vertical axis indicates angle / time (DEGREE×SEC), and the horizontal axis indicates time (FRAME). The graphs in Figures 45 and 46 show the relationship between the orientation of the mouse's body and distance, while the graphs in Figures 49 and 50 are calculated by dividing the values ​​in the graphs in Figures 45 and 46 by time. The graphs in Figures 45 and 46 quantified the change in angle of the mouse's body, but the graphs in Figures 49 and 50 differentiated the change in angle with time, and show the relationship between angle and angular velocity, similar to the relationship between speed and distance moved (amount moved) mentioned above, so by comparing these graphs with Figures 45 and 46, you can verify, for example, whether the mouse stopped at a moment. In other words, the graphs in Figures 45 and 46 allow you to see both the moving speed and the amount of movement at the same time. For example, if you see a number that jumps up and down from moment to moment, the higher the jump, the faster the mouse is rotating and moving. On the other hand, the lower the number (the closer to 0), the slower the mouse is rotating and moving.

[0160] Next, operations including detection of the outline of the mouse from a moving image and detection of the body orientation will be described with reference to Figs. Fig. 51 is a diagram showing a unit image at a certain timing of a video (original video) obtained by capturing images of the inside of a cage with a camera. Fig. 52 is a diagram showing the outlines (contours) of each of a plurality of mice (two mice) identified from the unit images. Fig. 53 is a diagram showing the orientation (direction from the center of the body to the tip of the nose) of one of the two mice. Fig. 54 is a diagram showing the orientation (direction from the center of the body to the tip of the nose) of the other of the two mice.

[0161] In the above-mentioned first and second embodiments, examples have been described in which the process of detecting the outer contour (outline) and the process of extracting body parts are performed separately, but by combining these processes, it is possible to analyze mouse behavior in more detail.

[0162] In this case, at a certain timing in the original video, a unit image as shown in Figure 51 is extracted, the extracted unit image is subjected to image analysis and individual identification, and then the outlines (contours) of each of multiple (2) white mice are output in different colors, for example, red and blue, as shown in Figure 52. Furthermore, as shown in Figure 53, the center of gravity of one of the two mice (the mouse in the upper outline (red) of Figure 52) is detected, and the direction of the body is detected from the relative positions of the ears, eyes, and nose tip. A line is then drawn from the center of gravity in the direction of the nose tip to detect the direction of the mouse's face (direction of the nose tip).

[0163] Similarly, as shown in Figure 54, the center of gravity of the other mouse (the mouse in the lower outline (blue) of Figure 52) is detected, and the orientation of the body is detected from the relative positions of the ears, eyes, and nose tip, and a line is drawn from the center of gravity to the nose tip, thereby detecting the orientation of the mouse's face (direction of the nose tip).

[0164] In this way, by combining the processes of contour recognition and part extraction, it is possible to detect the direction of the mouse's face (the direction of the nose tip) and perform a more detailed analysis of the mouse's behavior, such as turning, moving forward, or backing away while it remains in place.

[0165] In other words, the information processing device to which the present invention is applied can take various embodiments having the following configurations. The information processing device corresponds to the image processing device 2 having the functional configuration of Fig. 3 described in the first embodiment and the functional configuration of Fig. 24 described in the third embodiment. That is, an information processing device to which the present invention is applied (for example, the image processing device 2 in FIG. 3 and the server 200 in FIG. 24) An image acquisition means (e.g., image acquisition unit 51 in FIG. 3 ) for acquiring an image to be analyzed (e.g., a video or other image) that is obtained by capturing an image of one or more animals moving within a certain range of movement and that is composed of a plurality of unit images arranged in a time direction; a part extraction unit (e.g., part extraction unit 52 in FIG. 3 ) for extracting body parts (parts of bones and joints, center of gravity of a body, etc.) of each of the one or more animals using a skeleton estimation model that estimates and outputs a skeleton of an animal's body when a unit image is input; an individual identification means (e.g., the individual identification unit 54 in FIG. 3 ) for identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the one or more body parts of each of the one or more animals extracted from each of the plurality of unit images by the part extraction means, to an individual identification model which outputs an individual of the animal when a time series of the one or more body parts of the animal is input; A designation means (e.g., the designation unit 471 in FIG. 24 ) for designating analysis attributes for the image to be analyzed (the type of animal included in the image to be analyzed (e.g., mouse, rat, etc.), the imaging direction of the animal to be analyzed (e.g., upward, obliquely upward, sideways, etc.), and the color of the animal to be analyzed (e.g., white, black, etc.)); a model selection means (e.g., model selection unit 472 in FIG. 24, etc.) that selects an object to be applied to said part extraction means (e.g., part extraction unit 52 in FIG. 3) from among a plurality of types of said skeleton estimation models based on the analysis attribute of said image designated by said designation means, and selects an object to be applied to said individual identification means (e.g., individual identification unit 54 in FIG. 3) from among a plurality of types of said individual identification models; Equipped with. In an information processing device configured in this manner, multiple types of individual identification models or individual identification models are prepared in advance, and when analysis attributes for the image to be analyzed are specified, a target to be applied to the part extraction means (e.g., part extraction unit 52 in Figure 3) is selected from multiple types of skeletal estimation models based on the analysis attributes of the specified image, and a target to be applied to the individual identification means (e.g., individual identification unit 54 in Figure 3) is selected from multiple types of individual identification models. When an instruction to analyze an image is given, a skeletal estimation model and individual identification model for an animal selected from multiple types are used to analyze multiple parts extracted from each of the multiple unit images of the image in chronological order, and each individual of one or more animals is identified based on the analysis results, so that each individual mouse can be correctly identified from images captured of one or more animals (e.g., mice) active within a certain behavioral range. Furthermore, the information processing device (for example, the image processing device 2 in FIG. 3 and the server 200 in FIG. 24 ) A generating means (e.g., the data frame generating unit 53 of FIG. 3) for generating a data frame indicating a transition of the position of the part that changes in accordance with the behavior of the animal; a behavior determination unit (e.g., the behavior determination unit 56 in FIG. 3) that, when a transition in the position of the body part in the data frame satisfies one or more preset conditions, determines a behavior corresponding to the satisfied condition; It further comprises: By configuring it in this way, when the change in the position of the animal's body parts satisfies a condition, the behavior corresponding to that condition is judged, so that the reason for the behavior of each mouse moving within a certain movement range can be known. In other words, individual animals can be identified from video footage and their behavior can be evaluated based on the movements of each part of the body. The behavior determination means (for example, the behavior determination unit 56 in FIG. 3) a behavior detection means (e.g., the behavior detection unit 81 in FIG. 3 ) for detecting behavior related to a positional relationship between the part of the animal and another specific part (a watering hole, a feeding hole, a part of another animal) when a condition that specifies the positional relationship between the part of the animal and another specific part is satisfied; Further comprising: By doing so, we can learn the reasons for animals' behaviors, such as scratching behaviors that indicate stress while they are moving within a limited range of movement, and grooming behaviors toward other animals that indicate attempts to build relationships with them. The action, The behavior includes at least one of behaviors related to the sociality of each of the animals, behaviors related to interactions between animals that exist in the same home range, and behaviors related to relationships between animals. This makes it possible to know whether the reason for an animal's behavior is social behavior, behavior related to interactions between animals in the same habitat, or behavior related to relationships between animals. a marker image generating means (e.g., the marker image generating unit 55 in FIG. 3) for generating an image with markers in which the markers indicate the body parts of the animal extracted by the part extraction means (e.g., the part extraction unit 52 in FIG. 3) and the individual animals identified by the individual identification means (e.g., the individual identification unit 54 in FIG. 3) are associated with the body parts; Equipped with. By displaying an image with markers, one or more animals present in the image can be identified by the markers. After identifying the animals, the body parts of the animals can be detected, and their changes (body movements) can be assigned meaning and evaluated. A first information processing device (for example, the image processing device 2 in FIG. 3 or the server 200 in FIG. 24) An image acquisition means (e.g., image acquisition unit 51 in FIG. 3 ) for acquiring a group of a plurality of images (video) in which frame images each composed of a plurality of unit images (pixels) are arranged in a time direction, the group being obtained by capturing an image of one or more animals moving within a certain range of movement; A determination means (individual recognition unit 71 in FIG. 3 etc.) for determining to which region of each of the one or more animals each of the unit images belongs; A means for identifying the position of a specific part (eyes, nose, ears, legs, tail, skeleton, etc.) of the animal for each region of the animal by coordinates indicating a distance from a predetermined reference point in the image (e.g., a data frame generating unit 53 of FIG. 3 ); an individual identification means (e.g., the individual identification unit 54 in FIG. 3) that analyzes the state in which the position coordinates of the part identified by the identification means change over time, and identifies (classifies) each of the one or more animals included in the image based on the analysis result; Equipped with. and behavior determination means (e.g., behavior determination unit 56 in FIG. 3 ) for determining whether or not a transition in the coordinates of the part matches a preset condition for each animal identified (classified) by the individual identification means (e.g., individual identification unit 54 in FIG. 3 ), and assigning a behavior label of the animal corresponding to the matched condition (scratching behavior, sleeping, grooming behavior, etc.) to time information (time stamp) of the image including the part. It further comprises: This makes it possible to know whether the animal's behavior is a scratching behavior, a sleeping behavior, a grooming behavior, or the like. [Explanation of symbols]

[0166] 1 camera, 2 information processing device, 11 CPU, 41 image DB, 42 model DB, 43 material DB, 51 image acquisition unit, 52 part extraction unit, 53 data frame generation unit, 54 individual identification unit, 55 marker image generation unit, 56 behavior determination unit, 57 contour detection unit, 61 video acquisition unit, 62 unit image generation unit, 81 behavior detection unit, 82 behavior prediction unit, 71, 91...individual recognition unit, 72...part detection unit, 92...outer shell identification unit, 450...image analysis unit, 451...web service unit, 452...process unit, 461...authentication unit, 462...search unit, 463...search result output unit, 464...analysis data addition unit, 465...analysis result display control unit, 471...designation unit, 472...model selection unit, 491...upload data management unit, 492...unprocessed data management unit

Claims

1. an image acquisition means for acquiring an image to be analyzed, the image being obtained by capturing an image of one or more animals moving within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a part extraction means for extracting a part of a body of each of the one or more animals using a skeleton estimation model which estimates and outputs a skeleton of an animal's body when a unit image is input, for each of the plurality of unit images; an individual identification means for identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the one or more body parts of the animal extracted from each of the plurality of unit images by the part extraction means into an individual identification model which outputs an individual of the animal when a time series of the one or more body parts of the animal is input; A designation means for designating an analysis attribute for the image to be analyzed; a model selection means for selecting a model to be applied to the part extraction means from among a plurality of types of the skeletal estimation models based on the analysis attribute of the image designated by the designation means, and for selecting a model to be applied to the individual identification means from among a plurality of types of the individual identification models; a generating means for generating a data frame indicating a transition of the position of the part that changes in accordance with the behavior of the animal; and a behavior determination means for determining, when a transition of the position of the body part in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, The information processing device, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

2. The behavior determination means a behavior detection means for detecting a behavior related to a positional relationship between the part of the animal and another specific part when a condition that specifies the positional relationship is satisfied; The information processing device according to claim 1 , further comprising:

3. The action, At least one of the following behaviors is included: behaviors related to the sociality of each animal, behaviors related to interactions between animals in the same home range, and behaviors related to relationships between animals; The information processing device according to claim 2 .

4. An information processing method executed by an information processing device, an image acquiring step of acquiring an image obtained by capturing an image of one or more animals being active within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a part extraction step of extracting body parts of each of the one or more animals using a skeleton estimation model that estimates and outputs a skeleton of an animal's body when a unit image is input, for each of the plurality of unit images; an individual identification step of identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the one or more body parts of the animal extracted from each of the plurality of unit images into an individual identification model which outputs an individual of the animal when a time series of the one or more body parts of the animal is input; A designation step of designating an analysis attribute for the image to be analyzed; selecting a model to be applied to the part extraction step from among a plurality of types of the skeletal estimation models based on the analysis attribute of the specified image, and selecting a model to be applied to the individual identification step from among a plurality of types of the individual identification models; A generation step of generating a data frame showing a transition of the position of the part that changes in accordance with the behavior of the animal; and a behavior determination step of determining, when a transition of the position of the body part in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, An information processing method, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

5. A computer that controls an information processing device includes: an image acquiring step of acquiring an image obtained by capturing an image of one or more animals being active within a certain range of movement, the image being composed of a plurality of unit images arranged in a time direction; a part extraction step of extracting body parts of each of the one or more animals using a skeleton estimation model that estimates and outputs a skeleton of an animal's body when a unit image is input, for each of the plurality of unit images; an individual identification step of identifying each individual of the one or more animals in each of the plurality of unit images based on an output obtained as a result of inputting a time series of the one or more body parts of the animal extracted from each of the plurality of unit images into an individual identification model which outputs an individual of the animal when a time series of the one or more body parts of the animal is input; A designation step of designating an analysis attribute for the image to be analyzed; a model selection step of selecting a model to be applied to the part extraction step from among a plurality of types of the skeletal estimation models, and selecting a model to be applied to the individual identification step from among a plurality of types of the individual identification models, based on the analysis attribute of the specified image; A generation step of generating a data frame showing a transition of the position of the part that changes in accordance with the behavior of the animal; and a behavior determination step of determining, when a transition of the position of the body part in the data frame satisfies one or more preset conditions, a behavior corresponding to the satisfied condition, A program for executing a control process, wherein the analysis attribute is a type of the analysis object and / or an imaging direction of the analysis object.

Images