Information processing device, information processing method, and non-transitory recording medium

Abstract

The subject of the present invention is to be able to distinguish individual mice from an image obtained by photographing one or more mice moving within a certain range of motion. The image processing device (2) includes an image acquisition unit (51), a part extraction unit (52), and an individual identification unit (54). The image acquisition unit (51) acquires a dynamic image obtained by photographing a situation in which one or more animals move within a certain range of motion. The outline detection unit (57) detects the body outline of one or more animals for each of the multiple unit images included in the dynamic image. The individual identification unit (54) analyzes the multiple outlines detected by the outline detection unit (57) from each of the multiple unit images in a time series, and identifies the individual of one or more animals in each of the multiple unit images based on the analysis results. The subject is thus solved.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, and a program. Background Art

[0002] As a conventional technology, there is a technology for identifying individual animals such as cats and processing the biological information of the individual animals (for example, Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-007625 Summary of the Invention

[0006] (Problems to be solved by the invention)

[0007] However, although the prior art disclosed in the above-mentioned documents can measure the biological data of animals in a non-contact manner, it is unable to perform individual identification in the presence of continuously moving animals or multiple animals.

[0008] Furthermore, when processing individual biological information, conventional technology requires thermal imaging, etc., in addition to cameras, to confirm the health status of individually identified animals, which is costly. Furthermore, the aforementioned literature does not mention uses other than health status management.

[0009] The present invention has been made in view of such circumstances, and aims to be able to identify one or more mice moving within a certain range of motion.

[0010] (Technical solutions to solve problems)

[0011] In order to achieve the above-mentioned object, an information processing device according to one embodiment of the present invention includes:

[0012] An image acquisition unit is used to acquire an image of an analysis object, wherein the image of the analysis object is obtained by photographing one or more animals moving within a certain range of motion and is composed of a plurality of unit images arranged in a time direction;

[0013] An outline detection unit detects the body outline of each of the at least one animal using a skeleton estimation model for each of the plurality of unit images, wherein the skeleton estimation model estimates and outputs the body skeleton of the animal when the unit image is input;

[0014] an individual recognition unit for recognizing each individual of the one or more animals in each of the plurality of unit images based on an output obtained by inputting a time series of the body outlines of each of the one or more animals detected by the outline detection unit from each of the plurality of unit images into an individual recognition model, the individual recognition model outputting the individual of the animal when inputting the time series of the one or more body outlines of the animal;

[0015] a specifying unit configured to specify an analysis attribute of the image of the analysis object; and

[0016] The model selection unit selects an object suitable for the aforementioned outline detection unit from the plurality of aforementioned skeleton estimation models according to the aforementioned analysis attributes of the aforementioned image specified by the aforementioned specifying unit, and selects an object suitable for the aforementioned individual recognition unit from the plurality of aforementioned individual recognition models.

[0017] In this way, multiple skeleton estimation models and multiple individual identification models are prepared in advance, and according to the analysis attributes of the image specified by the specified unit, the skeleton estimation model of the object suitable for the outline detection unit is selected from the multiple skeleton estimation models, and the individual identification model of the object suitable for the individual identification unit is selected from the multiple individual identification models.

[0018] Then, through analysis instructions, for each of the multiple unit images included in the image, the body outlines of one or more animals are detected, the detected outlines are analyzed in time series, and the individual animals of one or more animals are identified based on the analysis results. In this way, individual animals can be distinguished from images obtained by photographing one or more animals moving within a certain range of motion.

[0019] An information processing method and a program corresponding to the above-mentioned information processing device according to one aspect of the present invention are also provided as an information processing method and a program according to one aspect of the present invention.

[0020] (Effects of the Invention)

[0021] According to the present invention, it is possible to identify one or more mice moving within a certain range of motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram showing a configuration example of an information processing system including an image processing apparatus according to one embodiment of the information processing apparatus of the present invention.

[0023] Figure 2 Yes Figure 1 A block diagram showing an example of the hardware configuration of an image processing device according to the information processing apparatus of the present invention in an information processing system.

[0024] Figure 3Yes Figure 1 The first embodiment of the information processing system is Figure 2 A functional block diagram of a first embodiment of the functional configuration of an image processing device.

[0025] Figure 4 Yes means having Figure 3 Flowchart of the operation of an image processing device with the functional structure of FIG.

[0026] Figure 5 This is a diagram showing an example of a unit image obtained from a video.

[0027] Figure 6 It means in Figure 5 An example of a unit image obtained from an image after a unit image of .

[0028] Figure 7 It means in Figure 6 An example of a unit image obtained from an image after a unit image of .

[0029] Figure 8 It means from Figure 5 A diagram showing the situation where an individual is identified from a unit image.

[0030] Figure 9 It means from Figure 8 A diagram showing the situation where parts are detected in a unit image.

[0031] Figure 10 It means Figure 9 Figure 4 shows the situation where the parts are extracted.

[0032] Figure 11 It means that Figures 5 to 7 A diagram showing the situation of tracking the parts extracted from the unit image.

[0033] Figure 12 Yes Figure 7 FIG is a diagram of a tracking image in which the unit image at the time t3 is marked.

[0034] Figure 13 This is a diagram showing an example of detecting the drinking behavior of mice.

[0035] Figure 14 It is a figure which shows the detection example of the eating behavior of a mouse.

[0036] Figure 15 This is a diagram showing an example of detecting the mutual interference behavior between mice.

[0037] Figure 16 Yes Figure 1 The second embodiment of the information processing system is Figure 2A functional block diagram of a second embodiment of the functional structure of an image processing device.

[0038] Figure 17 Yes means having Figure 16 Flowchart of the operation of an image processing device with the functional structure of FIG.

[0039] Figure 18 This is a diagram showing an example of a unit image obtained from a video.

[0040] Figure 19 It means in Figure 18 An example of a unit image obtained from an image after a unit image of .

[0041] Figure 20 It means from Figure 18 A diagram showing the situation where contours are detected in a unit image.

[0042] Figure 21 It means from Figure 19 A diagram showing the situation where contours are detected in a unit image.

[0043] Figure 22 This is a diagram showing an example of detecting scratching behavior.

[0044] Figure 23 It means Figure 1 A diagram outlining a business model for commercializing information processing systems.

[0045] Figure 24 Yes Figure 1 The third embodiment of the information processing system is to Figure 3 The functional configuration of the image processing device is a functional block diagram of the functional configuration of a server as an image analysis unit and a requester-side PC.

[0046] Figure 25 Yes Figure 24 Figure 1 shows the search screen displayed on a PC.

[0047] Figure 26 Yes Figure 25 Figure of the analysis data addition screen that pops up on the search screen.

[0048] Figure 27 Yes Figure 25 Figure 2 shows the dynamic image confirmation screen that pops up on the search screen.

[0049] Figure 28 This is a diagram showing an example of a report generated from a moving image.

[0050] Figure 29 This diagram shows an example of a configuration for forming a dedicated network connection with strong security.

[0051] Figure 30 This is a diagram showing the control panel screen of this information processing system.

[0052] Figure 31 This is a graph showing the position of the mouse in the cage and its movement trajectory from the start of measurement to 1320 frames (44 seconds) of initial movement.

[0053] Figure 32 It will Figure 31 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0054] Figure 33 This is a graph showing the position distribution of mice in each of nine areas (0 to 8) in which the floor surface of the cage is divided from the start of measurement to 1320 frames (44 seconds) of initial movement.

[0055] Figure 34 It will Figure 33 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0056] Figure 35 The bar graph shows the time the mouse stayed in each of the nine zones in which the cage floor was divided from the start of measurement to 1320 frames (44 seconds) of initial movement.

[0057] Figure 36 Will Figure 35 The chart then depicts a bar chart up to 18030 FRAME (approximately 10 minutes).

[0058] Figure 37 This is a graph showing the existence time of the mouse when the vicinity of the center of the cage (the fourth area) is distinguished from the other areas (near the edge) from the start of measurement to 1320 frames (44 seconds) of initial movement.

[0059] Figure 38 It will Figure 37 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0060] Figure 39 This is a graph showing the relationship between the period of 1320 frames from the start of measurement to the initial movement and the movement distance of the mouse per 1 second (30 frames).

[0061] Figure 40 It will Figure 39 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0062] Figure 41 This is a graph showing the relationship between the period of 1320 frames from the start of measurement to the initial movement and the instantaneous body orientation of the mouse in the cage.

[0063] Figure 42 It will Figure 41 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0064] Figure 43 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the total movement distance (total movement amount) of the mouse.

[0065] Figure 44 It will Figure 43 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0066] Figure 45 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the turning behavior (angular velocity) of the mouse.

[0067] Figure 46 It will Figure 45 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0068] Figure 47 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the speed of mouse movement.

[0069] Figure 48 It will Figure 47 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0070] Figure 49 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the angular velocity of mouse movement.

[0071] Figure 50 It will Figure 49 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0072] Figure 51 This is a diagram showing a unit image at a certain moment in a moving image (original moving image) obtained by photographing the inside of a cage with a camera.

[0073] Figure 52 This is a diagram showing the outlines (contours) of each of a plurality of (two) mice recognized from a unit image.

[0074] Figure 53 This diagram shows the orientation (the direction from the center of the body to the tip of the nose) of one of the two mice.

[0075] Figure 54 This diagram shows the orientation of the other mouse (the direction from the center of the body to the tip of the nose). DETAILED DESCRIPTION

[0076] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0077] Figure 1 This is a diagram showing the configuration of an information processing system including an image processing apparatus according to one embodiment of the information processing apparatus of the present invention.

[0078] (First embodiment)

[0079] Figure 1 The information processing system shown includes a camera 1 and an image processing device 2 connected to the camera 1 via a network N. The camera 1 is used to capture images of a room containing one or more mice ( Figure 1 In the example shown, cage C for mice (X1, X2) is set up (configured).

[0080] The network N includes not only wired networks but also wireless networks, etc. The cage C is a containment unit for allowing animals to move within a certain range of motion.

[0081] The camera 1 is, for example, a digital camera or a web camera that captures moving images, and outputs a moving image captured from above inside the cage C to the image processing device 2 .

[0082] Here, a moving image is an image composed of a plurality of unit images arranged in the time direction, and is also called a video. The unit image may be a field image, but here it is a frame image.

[0083] Image processing device 2 uses the moving image captured by camera 1 and a learning model (described in detail later) stored in model DB 42 to identify individual mice X1 and X2 included in the moving image as subjects. Image processing device 2 further detects factors (causes) underlying the behavior, such as the sociality of each mouse X1 and X2, the interactions between mice X1 and X2 within the same range of motion, and the relationships between mice X1 and X2, based on the behavioral patterns (habits, etc.) of the identified individual mice X1 and X2.

[0084] The image processing device 2 generates a tracking image and outputs it to an output unit 16 such as a display. The tracking image is given a mark such as a partial part of the body skeleton of one or more mice (tracking points) or an individual ID (Identification) included in the frame image extracted from the dynamic image, and at least one of the objects for visually identifying the outer contour of each mouse (and the surrounding contour).

[0085] If a printer is connected to the image processing apparatus 2 as the output unit 16 , the tracking image can also be printed.

[0086] The object is a contour line representing the outline of an individual mouse, a mask image, etc. The mark includes, for example, tracking points represented by graphics such as circles, triangles, and squares, and individual IDs represented by alphanumeric characters.

[0087] In addition, refer to Figure 3 The functional configuration and processing details of the image processing device 2 will be described later with reference to the following drawings.

[0088] Figure 2 Yes Figure 1 A block diagram showing an example of the hardware configuration of an image processing device according to the information processing apparatus of the present invention in an information processing system.

[0089] The image processing device 2 includes a CPU (Central Processing Unit) 11 , a ROM (Read Only Memory) 12 , a RAM (Random Access Memory) 13 , a bus 14 , an input / output interface 15 , an output unit 16 , an input unit 17 , a storage unit 18 , a communication unit 19 , and a drive 20 .

[0090] The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 18 to the RAM 13 .

[0091] The RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various processes.

[0092] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. The bus 14 is also connected to an input / output interface 15. The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0093] The output unit 16 is composed of a display, a speaker, etc., and outputs images and sounds.

[0094] The input unit 17 is composed of a keyboard, a mouse, and the like, and inputs various information in accordance with user's instruction operations.

[0095] The storage unit 18 is composed of a hard disk or the like, and stores data of various information.

[0096] The communication unit 19 controls the communication with other communication partners (eg Figure 1 Communication between cameras 1).

[0097] A removable medium 21 composed of a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted on the drive 20. Programs read from the removable medium 21 by the drive 20 are installed in the storage unit 18 as needed. Furthermore, similarly to the storage unit 18, the removable medium 21 can also store various data stored in the storage unit 18.

[0098] Figure 3 Yes Figure 1 The first embodiment of the information processing system is Figure 2 A functional block diagram of a first embodiment showing the functional configuration of an image processing device 2.

[0099] Figure 2 The storage unit 18 of the image processing device 2 shown stores an image DB 41 , a model DB 42 , and a material DB 43 .

[0100] The image DB 41 stores data of a moving image obtained from the camera 1, data of a plurality of frame images constituting the moving image (data of a still image), data of a tracking image obtained by assigning an object for animal tracking to the frame image, a mark representing an individual, etc., and a table (data frame) representing the transition of position information of the parts of the mice X1 and X2.

[0101] The model DB 42 stores a plurality of learning models. Specifically, the model DB 42 stores a plurality (one or more) of skeletal estimation models for estimating the skeleton of each animal (mouse, rat, etc.), a plurality (one or more) of individual recognition models for identifying each individual animal (if there are two identical animals, each animal is identified as a different individual), and a plurality (one or more) of mathematical models for determining the behavior of each individual animal.

[0102] The skeleton estimation model is designed to output a learning model of the animal's skeleton when an image of the animal is input.

[0103] The individual recognition model is designed to output a learning model of the individual animal when a time series of one or more parts of the animal's body is input.

[0104] A mathematical model is a model for analyzing animal behavior, and is designed to output information indicating the behavior of the animal when one or more images of the animal that change in time series are input.

[0105] The results of machine learning and updating pre-prepared moving and still images of each individual animal are stored as learning data. This data is then used to identify new moving or still images as animals or individuals, analyze the behavior of each individual, and output analysis results. This learning data is referred to as a "learning model" in this manual.

[0106] Specifically, based on the dynamic images obtained by filming the activities of a mouse as an example of an animal, a skeletal estimation model is used to estimate the configuration (skeleton) of the bones of the head, neck, arms, feet, tail, etc. of one or more mice in advance. Based on the position and movement of the feature points (body parts such as the mouse's eyes, nose, ears, toes, etc.) extracted from the skeleton of the animal, the pre-selected mouse is individually identified using an individual recognition model, and the result of the individual behavior determined by the mathematical model is output.

[0107] That is, the learning model in the first embodiment is a learning model that analyzes individual animals and their behaviors using techniques of skeleton estimation, individual recognition, and behavior estimation.

[0108] Furthermore, within the aforementioned model set, for example, within the skeleton estimation model, data from any pixel within the image (all pixels in the case of a CNN model) is used as input, and appropriate values ​​(nose coordinates, probabilities of nose coordinates) are calculated for each animal model. Furthermore, a CNN is a neural network centered around convolutional and pooling layers.

[0109] The learning models stored in the model DB 42 are reference learning models generated by machine learning in advance using one or more (plural) learning data by a learning device (not shown), and newly learned data may also be added.

[0110] Machine learning can be applied, for example, to convolutional neural networks. However, this is merely an example, and other machine learning methods can also be applied. Furthermore, the learning model is not limited to machine learning models; an identifier that identifies individual animals using a predetermined algorithm can also be used.

[0111] That is, the learning model only needs to be a model learned and generated in the following way: when an image is input, the skeleton, outline, and movement of each mouse included in the image match those of a known mouse, and its attributes are assigned and output.

[0112] The material DB 43 stores data used to determine the reasons for mouse behavior, sociality, and other aspects of mouse behavior identified by the learning model. The data in the material DB 43 correlates mouse behavior with the sociality derived from that behavior and relationships with other mice, and can be used to determine various habits, customs, and ecosystems of mouse behavior.

[0113] For example, it is data that can derive not only the movement of a mouse but also the habits of the mouse that makes such movement, such as its sociality and the relationship it has with other mice.

[0114] The material DB 43 stores data that maps the conditions that trigger mouse behavior to the behavioral patterns derived from those conditions. Specifically, the material DB 43 stores data that serves as judgment material, used by mathematical models to determine the nature of mouse behavior detected from images. Examples of such judgment material include data on exploratory behavior, eating and drinking, running and walking, sleeping, scratching, grooming, and fighting.

[0115] like Figure 3 As shown, the image processing device 2 includes an image acquisition unit 51 , a part extraction unit 52 , a data frame generation unit 53 , an individual identification unit 54 , a marker image generation unit 55 , and a behavior determination unit 56 .

[0116] The image acquisition unit 51 acquires images of one or more animals such as mice X1 and X2 moving within a certain range of movement.

[0117] The image acquisition unit 51 includes a moving image acquisition unit 61 and a unit image generation unit 62 .

[0118] The moving image acquisition unit 61 acquires an image (moving image) captured by the camera 1. The unit image generation unit 62 generates an image to be analyzed, consisting of a plurality of unit images arranged in the time direction, based on the image (moving image) acquired by the moving image acquisition unit 61, and stores the image in the image DB 41. Specifically, the unit image generation unit 62 generates a plurality of unit image (still image) groups (frame images) in units of frames from the moving image.

[0119] The part extraction unit 52 uses a skeletal estimation model selected from a plurality of skeletal estimation models in the model DB 42 to extract one or more body parts of each animal from each of the plurality of unit images. Specifically, the part extraction unit 52 sequentially reads a plurality of frame images from the image DB 41 and identifies the image region of the animal acting alone within the unit images included in each frame image. Specifically, the part extraction unit 52 separates the background portion of the frame image from the outline of the animal's body to identify the image region of the animal's body.

[0120] Parts of the body may be one or more feature points such as the left and right eyes, nose, left and right ears, left and right front toes, left and right back toes, tail tip, joints of bones or joints, and the center of gravity of the body.

[0121] In addition to the above-mentioned frame image, the unit image also includes one pixel, a plurality of pixel groups, and the like.

[0122] The part extraction unit 52 includes an individual identification unit 71 and a part detection unit 72 .

[0123] For each of the multiple unit images included in the frame image, the individual identification unit 71 identifies any unit image that has changed (moved) as part of a behavioral individual. Specifically, the individual identification unit 71 binarizes the unit image and identifies any color that differs from the background image as part of the behavioral individual. The individual identification unit 71 identifies any portion of the frame image that has changed color compared to the preceding and following frame images as a region boundary.

[0124] For each of the multiple unit images, the body part detection unit 72 superimposes the skeleton output by the skeleton estimation model on the area of ​​the unit image identified as part of the individual, extracts the animal's body part, and uses this part as a tracking point. At least one body part, such as the nose and left or right eyes, may be detected.

[0125] In addition, in this embodiment, the body parts of one or more animals are extracted from the image by combining the part extraction unit 52 including the individual identification unit 71 and the part detection unit 72 and the skeleton estimation model, but the part extraction model that outputs the animal's body parts when the unit image is input may be stored in the model DB42 in advance, and the animal's body parts may be extracted by inputting the unit image into the model DB42.

[0126] The data frame generating unit 53 converts the body parts extracted by the part extracting unit 52 into data frames.

[0127] The data frame generation unit 53 specifies the position of a specific part of the animal (eyes, nose, ears, feet, tail, bones, etc.) using coordinates representing the distance from a predetermined reference point in the image for each region of the animal.

[0128] Specifically, the data frame generation unit 53 generates a table that associates the body parts extracted by the part extraction unit 52 with the positional information representing the parts in a two-dimensional (plane) coordinate system (x-axis, y-axis, etc.) based on a reference point in the cage C included in the frame image or a portion of the behaving individual. The generated table is called a data frame. The x-axis is the horizontal axis of the two-dimensional (plane) coordinate system, and the y-axis is the vertical axis of the plane.

[0129] That is, the data frame generating unit 53 generates a data frame indicating the transition of the position of the aforementioned part that changes with the behavior of the animal.

[0130] Furthermore, when a spatial position is acquired as a tracking point, it becomes a three-dimensional (stereo) coordinate system (x-axis, y-axis, z-axis). The z-axis is the depth axis of the three-dimensional (stereo) coordinate system.

[0131] The individual identification unit 54 identifies each individual of one or more animals in each of the multiple unit images based on the output obtained by inputting the time series of the body parts of each of the more than one animal extracted from each of the multiple unit images by the part extraction unit 52 into the individual identification model selected from the multiple individual identification models of the model DB42.

[0132] Specifically, the individual identification unit 54 uses an individual identification model selected from multiple types for images, analyzes one or more parts extracted from each of the multiple unit images by the part extraction unit 52 in a time series, and identifies each individual of one or more mice X1, X2 in each of the multiple unit images based on the analysis results.

[0133] The individual recognition unit 54 analyzes the temporal shift of the position coordinates of the body parts and recognizes one or more animals included in the frame image (classified as different individuals) based on the analysis result.

[0134] That is, the individual identification unit 54 identifies which animal the one or more parts belong to and which animal individual has the parts.

[0135] Specifically, the individual identification unit 54 refers to the model DB42, analyzes the data frames generated by the data frame generation unit 53 in time series, identifies each individual of one or more mice X1, X2 in each of the multiple data frames based on the analysis results, and assigns the individual ID of the identified mice X1, X2 to a tracking point.

[0136] The individual identification unit 54 converts the positional relationship of each part of each individual at an arbitrary time point or the frequency of satisfying a certain condition into data, for example.

[0137] Here, certain conditions include, for example, the position of a certain part and other parts existing at predetermined coordinate positions (predetermined range) within a certain period, or the distance to other individual parts being zero or close to zero continuing for a certain period.

[0138] The individual recognition unit 54 instructs the marked image generation unit 55 to generate an image in which a mark representing the tracking point of the mice X1 and X2 identified by the individual recognition unit 54 and at least one of the objects used to visually identify the individuals of the mice X1 and X2 are superimposed on the unit image, and the obtained marked image (tracking image) is output to the output unit 16.

[0139] The marked image generating unit 55 generates a marked image in which the body parts of the mice X1 and X2 extracted by the part extracting unit 52 are associated with the marks indicating the individuals of the mice X1 and X2 identified by the individual identifying unit 54 .

[0140] Specifically, the marked image generation unit 55 generates a marked image (tracking image) by attaching a mark that allows visual identification of the individual mice X1 and X2 identified by the individual identification unit 54 to the unit image. The mark may be, for example, an individual ID (text) or a tracking point for each individual. The object may be an outline, a frame, or a mask image colored differently for each individual.

[0141] The behavior determination unit 56 determines the behavior of one or more mice X1 and X2 identified by the individual identification unit 54. Here, the behavior includes not only the behavior of each mouse X1 and X2 but also the related behavior of multiple mice X1 and X2 in a unit image.

[0142] That is, when the positional transition of a part in a data frame satisfies any one of one or more pre-set conditions, the behavior determination unit 56 determines the behavior corresponding to the satisfied condition.

[0143] Specifically, the behavior determination unit 56 determines whether the coordinate shift of the part of each animal identified (classified) by the individual identification unit 54 meets the pre-set conditions based on the mathematical model selected from the multiple mathematical models of the model DB42 and the material data of the material DB43, and assigns the behavior label of the animal corresponding to the conditions that meet the conditions (scratching behavior is "01", sleeping is "02", grooming behavior is "03", etc.) to the moment information (timestamp) or data frame of the image frame including the part.

[0144] Thus, for example, when two mice X1 and X2 are housed in a cage C and are in an active environment, if the frequency of contact between the nose (part) of one mouse X1 and the nose (part) of another mouse X2 exceeds a predetermined number of times within a certain period of time, it can be derived that the individuals are grooming behaviors in an attempt to establish a friendly relationship, or in the case of a male and a female, they are attempting to engage in breeding behavior, etc.

[0145] The behavior determination unit 56 includes a behavior detection unit 81 and a behavior prediction unit 82 .

[0146] When a condition defining a positional relationship between a part of an animal and other specific parts (a drinking place, a food place, or a part of another animal) is satisfied, the behavior detection unit 81 detects a behavior associated with the positional relationship.

[0147] Specifically, the behavior detection unit 81 refers to the model DB 42 and the material DB 43 to determine whether the positional relationship between the parts of mice X1 and X2 at any point in time in the data frame, or the positional relationship between mice X1 and X2 and other life-related parts (such as drinking fountains and food dispensers) in the cage C, meets the conditions of the material DB 43, and detects behaviors corresponding to the satisfied conditions. The behavior detection unit 81 detects at least one of the sociality of each mouse X1 and X2, the interaction between mice X1 and X2 within the same behavioral range, and the relationship between mice X1 and X2.

[0148] The behavior prediction unit 82 predicts how the mice X1 and X2 will live in the future based on the relationship between the mice X1 and X2 detected by the behavior detection unit 81 .

[0149] Then, refer to Figure 4 Image processing performed by the information processing device will be described. Figure 4 It means that Figure 3 Flowchart of an example of the flow of image processing performed by an information processing device having the functional configuration.

[0150] In the information processing system of the first embodiment, one or more mice X1 and X2 moving in a cage C are photographed by a camera 1, and their dynamic images are input into an image processing device 2. The image processing device 2 performs image processing as described below to assist in identifying each individual of the one or more mice X1 and X2 and to determine the personality, sociality, etc. of each mouse X1 and X2.

[0151] In step S11 , the image acquisition unit 51 acquires an image (eg, a dynamic image) obtained by photographing one or more animals moving within a certain range of motion and composed of a plurality of unit images arranged in a time direction.

[0152] In step S12 , the part extraction unit 52 extracts one or more body parts (eyes, nose, ears, feet, part of bones or joints, center of gravity of the body, etc.) of each of the plurality of unit images included in the acquired moving image.

[0153] In step S13 , the individual recognition unit 54 analyzes the parts extracted from each of the unit images by the part extraction unit 52 in time series, and recognizes each individual of one or more animals in each of the unit images based on the analysis result.

[0154] Identifying an individual means, for example, Figure 1and Figure 3 As shown, mice X1 and X2 are housed in a cage C. An individual ID such as "0" is assigned to a part of mouse X1 to identify it as individual "0," and an individual ID such as "1" is assigned to a part of mouse X2 to identify it as individual "1."

[0155] In step S14 , the behavior determination unit 56 determines the behavior of the one or more mice X1 and X2 identified by the individual identification unit 54 .

[0156] In the behavior determination unit 56, the behavior detection unit 81 detects at least one of the sociality of each mouse X1, X2, the interaction between mice X1, X2 in the same action range, and the relationship between mice X1, X2 based on the positional relationship between the parts of mice X1, X2 at any time.

[0157] In this way, according to the operation of the image processing device 2, partial parts of the body of each mouse X1, X2 are extracted from the dynamic image captured by the two mice X1, X2 moving in the cage C, the extracted parts are analyzed in time series, the individual mice X1, X2 are identified, and a tracking image is displayed in which an individual ID, tracking points and other marks, as well as objects for individual identification, are assigned to each identified individual. Therefore, the individual mice X1, X2 existing in the same image and in an active state can be distinguished.

[0158] In addition, regarding the behavior of mice X1 and X2, for example, the sociality of mice X1 and X2, the interaction between mice X1 and X2 in the same range of action, the relationship between mice X1 and X2, etc., the master-slave relationship (dominance, subordination, etc.), habits, ecology, etc. of mice X1 and X2 can be examined.

[0159] Next, refer to Figures 5 to 11 The image processing steps of the image processing device 2 in this information processing system will be described.

[0160] Figure 5 This is a diagram showing an example of a frame image obtained from a video. Figure 6 It means in Figure 5 An example of a frame image obtained from an image after the frame image of . Figure 7 It means in Figure 6 An example of a frame image obtained from an image after the frame image of . Figure 8 It means from Figure 5 A diagram showing a situation where an individual is identified in a frame image of . Figure 9 It means from Figure 8 A diagram showing the situation where parts are detected in a frame image. Figure 10 It means Figure 9 Figure 4 shows the situation where the parts are extracted. Figure 11 It means that Figures 5 to 8 A diagram showing a situation where the extracted parts in the frame image are tracked. Figure 12 Yes Figure 7 FIG is a diagram of a tracking image formed by marking the frame image at time t3.

[0161] In this information processing system, the interior of the cage C is photographed by a camera 1 , and the photographed moving images are sent to an image processing device 2 .

[0162] In the image processing device 2, the image acquisition unit 51 obtains the dynamic image input from the camera 1, such as Figures 5 to 7 As shown, frame images G1, G2, and G3 are acquired in time series in the order of time t1, t2, and t3.

[0163] Next, in the part extraction unit 52, the individual recognition unit 71 binarizes the unit images (pixels) included in the frame image G1 at time t1, for example, to separate the background (the cage portion) and the body parts of the white mice X1 and X2, and recognizes the regions by individual. In this example, Figure 8 As shown, a region 81 - 1 of the body part of the mouse X1 and a region 91 - 1 of the body part of the mouse X2 are recognized.

[0164] Then, if Figure 9 As shown, one or more parts 82-1, 92-1 of each region 81-1, 91-1 are detected by the part detection unit 72 from each region 81-1, 91-1 as tracking points for tracking behavior.

[0165] Figure 9 In the example shown in FIG1 , the part detection unit 72 calculates the centroid of the region 81-1 and detects the two-dimensional coordinates (x1, y1) of the part 82-1 representing the centroid of the region 81-1 in the frame image G1 as a tracking point. The part 82-1 is assigned "0" as the individual ID.

[0166] The center of gravity of the region 91-1 is similarly calculated, and the two-dimensional coordinates (x2, y2) of a part 92-1 representing the center of gravity of the region 91-1 in the frame image G1 are detected as a tracking point. The part 92-1 is assigned "1" as the individual ID.

[0167] In this manner, one or more parts of the animal are detected, and the movement of the body is digitized based on the parts obtained as a result of the detection.

[0168] In addition, as in the present embodiment, a method of finding the center of gravity of an area and using that position as a tracking point is one example. In addition, one or more of the animal's left and right eyes, nose, left and right front legs, left and right hind legs, joints of each leg, outline or center of the ear, spine, center of the tail, hairline, front end, etc. can also be detected as tracking points.

[0169] In this way, by detecting one or more parts of an animal, tracking the parts in a time series, and comparing with the learning model of model DB42, the animal having bones that move in this way can be determined, that is, the individual can be identified, for example, what animal the animal is.

[0170] Extract the body parts (tracking points) detected in this way, such as Figure 10 As shown, tracking data 101 including the extracted parts 82 - 1 and 92 - 1 and the individual ID is generated.

[0171] By also performing the above-mentioned processing of generating tracking data 101 on the frame images G2 and G3 and analyzing them in time series, the positional relationship of each part of each individual mouse X1 and X2 and the frequency of satisfying certain conditions can be digitized at any time point.

[0172] For example, Figure 11 As shown, by superimposing tracking data 101 for the parts generated corresponding to frame images G1 to G3, the trajectory of the animal's behavior can be obtained. Specifically, the trajectory of mouse X1 is obtained as vector data 111-1 and 111-2 representing the trajectories of movement of parts 82-1, 82-2, and 82-3. Furthermore, the trajectory of mouse X2 is obtained as vector data 112-1 and 112-2 representing the trajectories of movement of parts 92-1, 92-2, and 92-3.

[0173] In addition, the mark image generating unit 55 generates an image by superimposing a mark indicating a part of the animal, an individual ID, etc. on the unit image, so that Figure 12 Shown is a tracking image G3A.

[0174] In the tracking image G3A, for example, the frame image G3 at the time t3 (see Figure 7 ) is assigned a mark (circular mark) representing the site 82 - 3 of the mouse X1 , the individual ID “0”, and the trajectory line 120 .

[0175] Furthermore, the tracking image G3A displays a mark (triangle mark) indicating the part 92 - 3 of the mouse X2 , the individual ID “ 1 ”, and a trajectory line 121 .

[0176] By displaying the trajectory lines 120 and 121 starting from the time t1 and t2 of the frame images G1 and G2 for a predetermined time (eg, about 0.5 seconds) and reproducing the tracking image, the trajectory lines 120 and 121 can be kept displayed to reproduce the dynamic image of the movement of the mice X1 and X2.

[0177] Here, refer to Figures 13 to 15 This section describes several examples of determining the behavior of individual mice based on the positional relationships of one or more mouse parts, and examining their sociality, interactions, and relationships.

[0178] Figure 13 This is a diagram showing an example of detecting the drinking behavior of mice. Figure 14 It is a figure which shows the detection example of the eating behavior of a mouse. Figure 15 This is a diagram showing an example of detecting the mutual interference behavior between mice.

[0179] like Figure 13 As shown, in the image processing device 2, specific parts of the mouse (the nose and the left and right eyes in this example) are extracted as tracking points from the dynamic image of the cage containing the mouse captured by the camera 1, and the position of the tracking point (coordinates x, y) is tracked in time series. As a result, the coordinates representing the position of the nose and the left and right eyes of the mouse in each frame image are obtained for each individual mouse.

[0180] In the image processing device 2 , the behavior determination unit 56 determines the behavior of the animal based on whether or not the time-series change in the position of a predetermined part of the animal satisfies a predetermined condition.

[0181] Specifically, when a certain number of frames, e.g. Figure 13 In frames No. 150 to 156 (the part in brackets), when the distance between the mouse's nose position (coordinates) and the drinking water port position (coordinates) installed on the cage wall is zero or close to zero, the behavior determination unit 56 determines that the mouse is drinking water.

[0182] in addition, Figure 14 In frames No. 150 to 156 (the part in brackets), when the distance between the mouse's nose position (coordinates) and the position (coordinates) of the food arranged in the cage is zero or close to zero, the behavior determination unit 56 determines that the mouse is eating.

[0183] also, Figure 15 In frames No. 150 to 156 (the part in brackets), when the distance between the nose position (coordinates) of the mouse and the nose position (coordinates) of the mouse is zero or close to zero, the behavior determination unit 56 determines that the mice are interfering with each other.

[0184] As described above, according to the image processing device 2 in the information processing system of the first embodiment, partial body parts of each of the mice X1 and X2 are extracted from a dynamic image of the two mice X1 and X2 moving in the cage C, and the positions of the extracted parts are analyzed in a time series to identify the individual mice X1 and X2. Therefore, the individual mice X1 and X2 can be identified from the dynamic image obtained by capturing the mice X1 and X2 moving within a certain range of motion.

[0185] In addition, by judging the behavior of mice, such as their drinking behavior, eating behavior, and the mutual interference behavior of mice in the same range of action, it is possible to examine the habits and ecology of mice, the master-slave relationship between mice (dominance, subordination, etc.), etc.

[0186] In the individual recognition unit 54, on the premise that the position of the parts between the previous and next frames is not far apart, the individual can be correctly recognized based on the continuity of the position of the parts extracted in the previous stage.

[0187] The individual recognition unit 54 recognizes an individual based on the outline and positional information of the parts, and therefore can accurately recognize an individual regardless of the brightness of the shooting environment or the background.

[0188] Next, refer to Figure 16 The second embodiment will be described. Figure 3 The same functional configurations as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0189] Figure 16 Yes Figure 1 The second embodiment of the information processing system is Figure 2 A functional block diagram of a second embodiment of the functional configuration of an image processing device 2 is shown.

[0190] The image processing device 2 includes an outline detection unit 57. The outline detection unit 57 detects the body outline (body contour, etc.) of one or more animals for each of a plurality of unit images using a skeleton estimation model selected from a plurality of skeleton estimation models in the model DB 42.

[0191] The outline detection unit 57 includes an individual recognition unit 91 and an outline identification unit 92 .

[0192] The individual identification unit 91 identifies, for each of the plurality of unit images, a region including a unit image that has changed (moved) as one behavioral individual.

[0193] Specifically, the individual identification unit 91 binarizes each unit image and identifies a portion having a different color from the background image as a portion of the behavioral individual. Preferably, a portion having a different color from the frame image before and after in time series is used as a region boundary.

[0194] The outline specifying unit 92 specifies, for each of the plurality of unit images, a region including a portion recognized as one behavioral individual as the body outline (body contour, etc.) of each of the one or more mice X1 and X2.

[0195] The data frame generating unit 53 generates a data frame indicating the probability that the outline change accompanying the behavior of the animal is a specific behavior (see Figure 22 ).

[0196] The individual identification unit 54 identifies each individual of one or more animals in each of the multiple unit images based on the output obtained by inputting the time series of the body outlines of one or more animals detected by the outline detection unit 57 from each of the multiple unit images into the skeleton estimation model of the model DB42.

[0197] The individual recognition unit 54 analyzes the one or more outlines detected by the outline detection unit 57 from each of the plurality of unit images in a time series manner, and recognizes each individual of one or more animals in each of the plurality of unit images based on the analysis result.

[0198] Specifically, the individual identification unit 54 identifies each of the mice X1 and X2 by analyzing one or more outlines detected from each of the plurality of frame images in a time series with reference to the model DB 42, and outputs the individual information (such as the individual ID and the position of the outline) of the mice X1 and X2 obtained as a result of the identification to the behavior determination unit 56. In other words, the individual identification unit 54 identifies which of the two mice X1 and X2 in the cage the detected outline belongs to.

[0199] The learning model of the model DB42 in the second embodiment performs mechanical learning on the dynamic images and still images of each individual mouse prepared in advance, and stores the completed learning data, which is used to identify the mouse and output the individual information of the identification result (individual ID and the position of the outline, etc.) when a new dynamic image or still image is input.

[0200] Specifically, for example, if a dynamic image obtained by shooting the activities of each mouse is input into the model DB42, the individual ID of one or more mice included in the dynamic image and the position of the outline (body contour) of the mouse with the individual ID in the dynamic image will be output.

[0201] That is, by using the learning model in the second embodiment, a learning model using a skeleton estimation method is obtained, and the outline position of each individual can be detected from the region of each identified individual.

[0202] The behavior determination unit 56 digitizes the general behavior of each individual at any time or the frequency of meeting certain conditions based on the individual information input from the individual recognition unit 54 , and compares the individual behavior or conditions with the data in the material DB 43 .

[0203] Specifically, when the probability value of a specific behavior in the data frame satisfies any one of one or more pre-set conditions, the behavior determination unit 56 determines that the behavior corresponds to the satisfied condition.

[0204] Here, one or more conditions refer to the following conditions: for example, after one mouse performs a general behavior of curling up, another mouse performs a general behavior of snuggling up to the curled-up mouse, and the frequency of this behavior is several or more times.

[0205] The behavior determination unit 56 refers to the material DB 43 and determines that the relationship between the mice is good and that the mice are paired when the behavior of the mice identified by the individual identification unit 54 satisfies the condition.

[0206] As described above, according to the functional configuration of the image processing device 2 in the information processing system of the second embodiment, in addition to the same effects as those of the first embodiment, the following effects can be obtained.

[0207] That is, from a dynamic image captured of two mice X1 and X2 housed in a cage C and in motion, the body outlines of each mouse X1 and X2 are detected, the detected outlines are analyzed in time series, and each individual mouse X1 and X2 is identified. Therefore, each individual mouse X1 and X2 that is present in the same image and is in an active state can be distinguished.

[0208] (Second embodiment)

[0209] Next, refer to Figure 17 Description by Figure 16 The image processing operation performed by the information processing device having the functional structure of the second embodiment of the present invention. Figure 4 The same actions as those in the first embodiment are marked with Figure 4 The same steps are numbered and their descriptions are omitted. Figure 17 It means that Figure 16 Flowchart of an example of the flow of image processing performed by an information processing device having the functional configuration.

[0210] In the second embodiment, when a moving image is acquired from the camera 1 in step S21, in step S22, the outline detection unit 57 detects the body outline (body contour, etc.) of one or more animals for each unit image of a plurality of frame images included in the acquired moving image.

[0211] In step S23, the individual recognition unit 54 analyzes the outlines extracted from each of the plurality of unit images by the outline detection unit 57 in a time series manner, and recognizes each individual of one or more animals in each of the plurality of unit images based on the analysis results. The processing after step S24 is the same as that in the first embodiment ( Figure 4 )same.

[0212] According to the operation of the image processing device 2 having the functional structure of the second embodiment, the body outlines of each mouse X1 and X2 are detected for each unit image of the multiple frame images included in the dynamic image obtained from the camera 1, and the changes in the outlines over time accompanying the behavior of the mice X1 and X2 are analyzed, thereby identifying the individual mice X1 and X2, and thus the individual mice X1 and X2 reflected in the same image can be distinguished.

[0213] In addition, by monitoring the changes in the outlines of the identified mice X1 and X2 over time, the specific behaviors of the mice X1 and X2 are detected by satisfying predetermined conditions, thereby discovering not only existing ecosystems but also new ecosystems.

[0214] Next, refer to Figures 18 to 22 The image processing steps of the information processing device in this information processing system will be described.

[0215] Figure 18 This is a diagram showing an example of a frame image obtained from a video. Figure 19 It means in Figure 18 An example of a frame image obtained from an image after the frame image of . Figure 20 It means in Figure 18 FIG. 1 is a diagram showing a tracking image in which an object and individual labels indicating outlines are added to a frame image at time t4. Figure 21 It means in Figure 19 FIG. 1 is a diagram showing a tracking image in which an object and individual labels indicating an outline are added to the frame image at time t5. Figure 22 This is a diagram showing an example of detecting scratching behavior.

[0216] In addition, in the description Figures 18 to 21In this description, the mouse included in the frame image G4 at time t4 is referred to as mouse X1-1, and the mouse included in the frame image G5 at time t5 is referred to as mouse X1-2. The same applies to the other mice X2-1 and X2-2.

[0217] In this information processing system, the interior of the cage C is photographed by a camera 1 , and the photographed moving images are sent to an image processing device 2 .

[0218] In the image processing device 2, the image acquisition unit 51 acquires the dynamic image input from the camera 1 in a time series. Figure 18 The frame image G4 at time t4 shown, and Figure 19 Frame image G5 at time t5 is shown.

[0219] Next, the outline detection unit 57 uses the individual identification unit 91 to binarize the unit image included in the frame image G4 at time t4, for example. This creates regions of different colors for the background (the cage) and the body of the white mouse, and the regions are identified for each individual mouse. As a result, the body regions of each of the two mice are identified.

[0220] Figure 20 A display example of the tracking image G4A after individual recognition is shown. In this display example of the tracking image G4A, a blue object 121-1 is displayed on the outline (body outline) of the mouse X1-1, and a red object 122-1 is displayed on the outline (body outline) of the mouse X2-1.

[0221] Furthermore, in the tracking image G4A, “mouse 0.996” is displayed as an individual ID for identifying the individual mouse X1 - 1 , and a mark such as a frame 131 - 1 surrounding the individual is displayed.

[0222] In the tracking image G4A, “mouse 0.998” is displayed as an individual ID for identifying the individual mouse X2 - 1 , and a mark such as a frame 132 - 1 surrounding the individual is displayed.

[0223] In the frame image G5 at time t5, the unit image (pixel) is similarly binarized to segment the background (cage portion) and the body portion of the white mouse, and the region is identified for each individual.

[0224] Figure 21 In the example of the tracking image G5A after individual recognition, a blue object 121 - 2 is displayed on the outline (body outline) of the mouse X1 - 2 , and a red object 122 - 2 is displayed on the outline (body outline) of the mouse X2 - 2 .

[0225] Furthermore, in the tracking image G5A, “mouse 0.996” is displayed as an individual ID for identifying the individual mouse X1 - 2 , and a mark such as a frame 131 - 2 surrounding the individual.

[0226] In addition, in the tracking image G5A, “mouse 0.998” is displayed as an individual ID for identifying the individual mouse X2-2, and a mark such as a frame 132-2 surrounding the individual is displayed.

[0227] In this way, the outlines of the body parts of the two mice are detected, the individual mice are identified based on the changes in the outlines over time, and the specific behavior of the mice is determined when each individual meets the predetermined conditions at any time point, so that the reasons for the behavior of each mouse can be known.

[0228] Here, refer to Figure 22 This section describes an example of determining the behavior of individual mice based on the positional relationship of their outlines, and detecting their sociality, interactions, and relationships.

[0229] Figure 22 This is an example of detecting scratching behavior.

[0230] from Figure 22 Frame images are acquired sequentially starting from frame No. 1, and after individual identification of the mouse, the behavior determination unit 56 calculates the probability of whether the mouse is performing a specific behavior for each frame image, and frames the probability obtained as a calculation result for each frame image.

[0231] Then, based on the probability of the data frame, the behavior of the mouse is labeled (classified), and its identification information is assigned to the data frame and stored in the image DB 41 .

[0232] For example Figure 22 When the calculated probability in the data frames of frames No. 4502 to 4507 exceeds a certain threshold, the behavior determination unit 56 determines that the mouse has performed a specific behavior and assigns identification information representing the specific behavior (such as prediction (predict) "1", etc.) to the data frames of frames No. 4502 to 4507. At the same time, when the time series change of the mouse's outline (the behavior of the mouse) meets the conditions pre-set in the material DB43, it is determined to be a specific behavior of the mouse derived from the condition, such as the behavior of the mouse scratching the floor of the cage, that is, scratching behavior.

[0233] According to the image processing device 2 of the second embodiment, the probability of whether the mouse identified by the individual is performing a specific behavior is calculated for each frame image, and the data frame at the time when the mouse performs the specific behavior is marked based on the probability obtained as a result of the calculation. Therefore, not only the established behavior of the mouse can be detected, but also unexpected behavior.

[0234] As a result, each of multiple mice housed in a cage can be identified and observed, and common and novel behaviors performed by each mouse can be discovered.

[0235] (Third embodiment)

[0236] Then, refer to Figures 23 to 30 A third embodiment will be described.

[0237] In the above-mentioned first and second embodiments, models are only constructed for specific animal species, behaviors (mice, scratching behaviors, etc.), and shooting environments, and are limited to output in the form of RAW data (position information of various parts, etc.) and construction of condition judgment programs. However, in this third embodiment, the application is expanded to multiple animal species, behaviors, and shooting environments, thereby improving practicality.

[0238] In the third embodiment, annotations are performed for obtaining RAW data (teacher data is created for determining the location of the acquisition part), and RAW data is used to construct various behavior judgment conditions (mathematical models), program and modularize for application, and expand programs for detecting the movement, eating, and drinking of mice.

[0239] In addition, in the above-mentioned first and second embodiments, the minimum necessary tasks are achieved in their own hardware and software environments, but in the third embodiment, in order to improve versatility and enable general users with low IT literacy to use it stably, a user interface environment (hereinafter referred to as the "UI environment") that is consistent with business processes and easy to operate is constructed.

[0240] In addition, it is assumed that the system's execution environment also runs stably regardless of the conditions on the user (requester) side, and the specifications and costs are optimized according to the frequency and load of the executed tasks, thereby building a cloud service, that is, a system that serves as a server-client system.

[0241] First, refer to Figure 23 An overview of the information processing system according to the third embodiment will be described.

[0242] Figure 23 This is a third embodiment of the information processing system, and is a diagram showing an outline of a business model for commercializing the information processing systems of the first and second embodiments.

[0243] like Figure 23 As shown, the information processing system of the third embodiment is configured such that a device on the client 300 side of the requester Y communicates with the server 200 of the contractor who accepts the request.

[0244] After receiving the request from requester Y, server 200 analyzes the dynamic image uploaded by requester Y and sends the analysis result to requester Y.

[0245] The equipment on the client 300 side includes a camera 310 constituting an imaging environment of an animal, a cage 311 , and a client computer 320 (hereinafter referred to as “PC 320 ”) that collects dynamic images of an analysis target captured by the camera 310 .

[0246] The cage 311 is a container for animals to move within a certain range. The camera 310 captures the animals moving within the cage 311. The environment in which the animals are captured can be changed appropriately according to the request content.

[0247] PC 320 acquires the moving image (image) data captured by camera 310 and stores it locally (in an internal memory, etc.) PC 320 uploads the locally stored moving image data 321 to server 200 and requests analysis of the behavior of the animal included in the moving image.

[0248] In response to the request, the PC 320 acquires the analysis result information (CSV file 330 ) from the server 200 and the processed moving image data (processed moving image 331 ) obtained by adding a mark M to the position of the animal's eye.

[0249] The PC 320 creates data (chart 322 ) for analyzing the data in the CSV file 330 , or processes the dynamic image 331 as a part of the analysis data.

[0250] The server 200 includes an image analysis unit 450 , a network service unit 451 , a storage 410 , and a data warehouse 411 (hereinafter referred to as “DWH 411 ”).

[0251] The image analysis unit 450 includes the image analysis unit 450 shown in the first embodiment. Figure 3 The functional configuration shown in the second embodiment Figure 16 Functional composition, and performs image analysis processing.

[0252] The network service unit 451 has functions for authenticating login information input from the PC 320, searching for analysis targets, outputting search results, adding analysis data, and displaying analysis results. Specific functional configurations will be described later.

[0253] Furthermore, the network service unit 451 has functions such as encrypted communication, access source IP communication, and dedicated line communication, thereby realizing security measures.

[0254] That is, the network service unit 451 realizes an interface with the PC 320 .

[0255] The memory 410 stores moving image data requested for analysis by the network service unit 451 and moving image data being analyzed.

[0256] The DWH 411 stores various related data and systematically archives the results of collaborative processing with the data in the memory 410. This allows data to be reused and reduces the number of animal experiments.

[0257] Here, the data processing procedure (flow) of the information processing system according to the third embodiment will be described.

[0258] In step S101 , the animal's movements in the cage 311 are captured in the form of dynamic images, and the dynamic image data 321 is stored locally on the PC 320 .

[0259] In step S102 , when the contractor is requested to perform a dynamic image analysis, the dynamic image data 321 stored in the PC 320 is uploaded to the server 200 .

[0260] In the server 200 , the network service unit 451 stores the moving image data 321 uploaded from the PC 320 in the memory 410 .

[0261] In addition, in steps S101 and S102, the captured dynamic image data 321 is temporarily stored in PC320 and then uploaded to server 200. In the case of dynamic images captured continuously for 24 hours, the dynamic image data 321 captured by camera 310 can also be directly uploaded to server 200 as in step S103.

[0262] In step S104, in the server 200, the image analysis unit 450 reads the dynamic image data stored in the memory 410, performs analysis of the requested dynamic image data 321, saves the processed dynamic image 331 generated by processing the dynamic image during the analysis and the CSV file 330 of the analysis results in the memory 410, and outputs the analysis completion notification to the PC 320.

[0263] The PC 320 that has received the analysis completion notification displays a search screen. When the dynamic image data to be analyzed is specified, the network service unit 451 searches for the specified dynamic image data in step S105 and downloads the processed dynamic image 331 and the CSV file 330 of the analysis results to the PC 320 .

[0264] In step S106 , the PC 320 creates a graph 322 for analysis using the processed moving image 331 downloaded from the server 200 and the CSV file 330 of the analysis result, and attaches the processed moving image 331 as a supporting material to the graph 322 .

[0265] In addition, in the above-mentioned steps S105 and S106, the CSV file 330 (numerical data) of the analysis results is downloaded to PC320, and a chart 322 is created based on the CSV file 330. However, in addition to this, for example, as in step S107, a chart 322 can be created based on the CSV file 330 in the server 200 according to the request content, and the chart 322 as the analysis result can be downloaded to PC320.

[0266] Then, refer to Figure 24 right Figure 23 The functional configuration of the information processing system according to the third embodiment will be described. Figure 24 Yes Figure 23 A functional block diagram of the functional configuration of an information processing system according to a third embodiment of the present invention.

[0267] In addition, when explaining the functional configuration of the information processing system of the third embodiment, Figure 3 The functional configuration of the first embodiment shown, and Figure 16 The same functional configurations as those in the second embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0268] like Figure 24 As shown, an area of ​​the storage unit 18 of the server 200 stores an authentication DB 44. The authentication DB 44 stores authentication information for the requester Y to log in to the server 200. The authentication information is identification information for identifying the requester Y, such as a login ID and a password.

[0269] Furthermore, the model DB 42 pre-stores various learning models generated or updated through machine learning and other means. Examples of these learning models include skeleton estimation models, individual identification models, and mathematical models. These various learning models are configured, for example, for each animal species, image orientation, or animal color.

[0270] When executed with Figure 23 During the processing corresponding to steps S101 to S107 , the image analysis unit 450 , the network service unit 451 , and the processing unit 452 function in the CPU 11 of the server 200 .

[0271] The image analysis unit 450 uses a model selected for image data designated in the moving image data 321 acquired in response to a request from the requester Y to perform image analysis processing.

[0272] The image analysis unit 450 reads the image of the analysis object specified by the designation unit 471 from the image DB41, and uses the learning model (skeleton estimation model, individual recognition model, mathematical model, etc.) selected from multiple learning models by the model selection unit 472 and extracted from the model DB42 to perform analysis processing on the image.

[0273] The image processing performed by the image analysis unit 450 is processing such as individual recognition, outline recognition, and behavior analysis of animals in moving images described in the first and second embodiments.

[0274] When the requester Y logs in to the server 200 , the network service unit 451 performs login authentication, searches for the dynamic image analysis results requested by the requester Y, outputs the search results, adds analysis data, and controls the display of the analysis results.

[0275] The network service unit 451 includes an authentication unit 461 , a search unit 462 , a search result output unit 463 , an analysis data addition unit 464 , an analysis result display control unit 465 , and the like.

[0276] The authentication unit 461 performs user authentication by comparing the input login information with the authentication information in the authentication DB. If the information matches, the authentication unit 461 permits the requester Y to log in to the server 200.

[0277] The search unit 462 displays the search screen 251 (see Figure 25 ), according to the search request from the search screen 251, the image DB 41 is searched for analysis results, and the search results are transmitted to the search result output unit 463. On the search screen 251, a list of search results, addition of analysis data, display of analysis results, etc. can be performed.

[0278] The search result output unit 463 outputs the search results obtained by the search unit 462 to the PC 320 , and displays the results in a list on the search screen 251 of the PC 320 .

[0279] The analysis data adding unit 464 displays the analysis data adding screen 261 (see Figure 26 ).

[0280] By performing the analysis data addition operation from the analysis data addition screen 261 , the moving image data to be analyzed is uploaded to the image processing device 2 . In the image processing device 2 , the uploaded moving image data to be analyzed is additionally registered in the image DB 41 .

[0281] After uploading, analysis conditions can be specified for the new file added to the image DB 41 on the analysis data addition screen 261 .

[0282] The analysis data adding unit 464 includes a specifying unit 471 and a model selecting unit 472 .

[0283] The designation unit 471 designates analysis attributes of the image of the analysis target (type of analysis target (e.g., mouse, rat, etc.), shooting direction of the analysis target (e.g., top, obliquely top, horizontal, etc.), color of the analysis target (e.g., white, black, etc.)).

[0284] Specifically, the designation unit 471 displays Figure 26 The analysis data addition screen 261 is displayed.

[0285] A file column and a setting column are provided in the analysis data addition screen 261. The file column contains an icon of a file of an image to be analyzed and a bar graph indicating the progress of the analysis (progress status).

[0286] The setting bar is provided with radio buttons (specify buttons) for the requester Y to specify analysis attributes such as the type of animals included in the image of the analysis object (such as mice, rats, etc.), the shooting direction of the animal of the analysis object (such as above, diagonally above, horizontally, etc.), and the color of the animal of the analysis object (such as white, black, etc.).

[0287] When the requester Y uploads the image to be analyzed to the image processing apparatus 2 , the icon of the uploaded image file to be added is displayed in the file column of the analysis data addition screen 261 .

[0288] Then, requester Y uses the radio button in the setting column corresponding to the icon of the image file to set the image (the type of animal included in the image of the analysis object, the shooting direction of the animal of the analysis object, the color of the animal of the analysis object, etc.), thereby improving the image analysis accuracy during image analysis, and correctly identifying the animal type or individual included in the image, analyzing behavior, etc.

[0289] The model selection unit 472 selects a suitable skeleton estimation model from a plurality of skeleton estimation models stored in the model DB according to the analysis attribute of the image specified by the specifying unit 471. Figure 3 The part extraction unit 52 or Figure 16 The model of the outline detection unit 57 is selected, and a model suitable for the individual recognition unit 54 is selected from the plurality of individual recognition models stored in the model DB.

[0290] The analysis result display control unit 465 displays the analysis result CSV file 330 , the processed moving image 331 , and the like downloaded from the server 200 .

[0291] The processing unit 452 manages analytical image data and unprocessed data.

[0292] The processing unit 452 includes an upload data management unit 491 and an unprocessed data management unit 492 .

[0293] The upload data management unit 491 moves the image data uploaded from the PC 320 to the image DB 41 and updates the management file. The upload data management unit 491 also has a function of starting up every minute based on cron and a function of preventing duplicate starting.

[0294] Unprocessed data management unit 492 identifies unprocessed data in the management file. It copies the unprocessed data. It performs AI processing on the data in the processing directory. It stores the result file in the corresponding directory. It creates a CSV file 330 containing the analysis results and stores it in the corresponding directory. Similar to uploaded data management unit 491, unprocessed data management unit 492 includes functions such as a cron-based minute-by-minute startup function, a function to prevent duplicate startup, and a management file update function.

[0295] When executed with Figure 23 During the processing corresponding to steps S101 to S107 , in the CPU 141 of the PC 320 , the login management unit 421 , the screen control unit 422 , the analysis request unit 423 , and the analysis result display control unit 424 function.

[0296] The login management unit 421 displays a login screen on the PC 320 , transmits the login information input by the requester Y on the login screen to the server 200 , and requests login authentication.

[0297] Screen control unit 422 displays a search screen on PC 320, transmits the search keyword entered by requester Y into the search screen to server 200, and requests a search. The search screen then displays the search results for the request. The search screen displays a list of search results. Furthermore, analysis data can be added to the search screen, and analysis results can be displayed.

[0298] The analysis request unit 423 displays the analysis data addition screen on the PC 320 , transmits the file and analysis conditions specified by the requester Y on the analysis data addition screen to the server 200 , and requests analysis.

[0299] The analysis result display control unit 424 displays the analysis result display screen on the PC 320 , and displays the analysis result CSV file 330 and the processed moving image 331 downloaded from the server 200 on the analysis result display screen.

[0300] Below, refer to Figures 25 to 27 The operation of the information processing system according to the third embodiment will be described.

[0301] Figure 25 This figure shows a search screen displayed on a PC. Figure 26This figure shows the analysis data addition screen that pops up on the search screen. Figure 27 This figure shows a dynamic image confirmation screen that pops up on the search screen.

[0302] Figure 25 The search screen shown is displayed on PC320 (refer to Figure 24 The search screen 251 includes an input field for inputting search conditions such as a keyword and a period, an analysis data list, an analysis data add button, a dynamic image confirmation button, a CSV output button, and the like.

[0303] The analysis data list lists analysis data by file name, user, date and time, and analysis. The analysis item indicates the current analysis status, such as "Analysis Completed" or "In Progress." Select buttons are located on the left side of the analysis data list for selecting individual analysis data. By pressing these buttons, you can add analysis data, view dynamic images, and export to a CSV file.

[0304] In the search screen 251, when multiple analysis data are logged in to the server 200, the analysis data is filtered using the search conditions and displayed in the analysis data list. After selecting (specifying) the analysis data using the selection button, any one of the analysis data addition button, dynamic image confirmation button, and CSV output button is pressed to perform the processing corresponding to the button operation (analysis data addition, dynamic image confirmation, CSV output, etc.).

[0305] (Analysis data added)

[0306] exist Figure 25 In the search screen 251, after selecting (designating) the desired analysis data using the selection button, for example, when the analysis data addition button is pressed, Figure 26 The analysis data adding screen 261 shown is displayed as a pop-up on the search screen 251 .

[0307] The analysis data addition screen 261 includes files of analysis data to be added, analysis property setting fields for setting (designating) the type of moving images included in each file, a return button, a cancel button, a register button, and the like.

[0308] The analysis attribute setting column includes buttons for selecting, for example, the type of animal, the direction in which the animal is photographed, the color of the animal, and the like, and settings (designations) can be made using each button.

[0309] On the analysis data addition screen 261 , by selecting and setting a file to be analyzed for the moving image data uploaded as new analysis data, analysis of the subsequent analysis data can be automatically executed.

[0310] At this time, based on the type of animal set as analysis attributes in the analysis data additional screen 261, the shooting direction of the animal, the color of the animal, etc., a learning model of the animal used for the analysis object is selected from the multiple learning models (skeleton estimation model, individual identification model and mathematical model) stored in the model DB42.

[0311] (Dynamic image confirmation)

[0312] exist Figure 25 In the search screen 251, after selecting (designating) the desired analysis data using the selection button, for example, if the dynamic image confirmation button is pressed, Figure 27 The moving image confirmation screen 271 shown is displayed as a pop-up on the search screen 251 .

[0313] The moving image confirmation screen 271 includes a playback area for playing back the added analysis data, a return button, a download button, and the like. The playback area includes an image (still image) where the selected analysis data (moving image data) is paused, and a playback button (triangle icon). Clicking the playback button (triangle icon) starts the still image moving and plays it back as a moving image.

[0314] In the moving image confirmation screen 271 , when it is difficult to distinguish whether the moving image data uploaded as new analysis data is the analysis target data or the processed moving image after analysis, the moving image can be reproduced for confirmation.

[0315] Furthermore, by pressing the download button, the analysis data displayed in the playback area can be downloaded. This download function is effective when it is desired to use the processed moving image 331 on the PC 320 side.

[0316] (Advantages of using this information processing system)

[0317] Reference Figure 28 Advantages of using this information processing system are described.

[0318] Figure 28 This is a diagram showing an example of a report generated from a moving image.

[0319] This information processing system can output detailed quantitative data on the opening and closing of mouse eyes, which cannot be confirmed visually, making it possible to confirm previously unattainable content with unprecedented accuracy.

[0320] For example, Figure 28As shown, when dynamic image data 281 of a mouse is uploaded from the PC 320 of the requester Y to the server 200 and a request is made to analyze the condition of the mouse's eyes, a mark M is set at the position of the mouse's eyes by the part detection unit 72 to identify the situation where the area of ​​the mouse's eyes becomes wider when the eyes are open and becomes narrower when the eyes are closed, that is, the opening and closing of the eyes are identified, so that a graph 283 showing the changes in the area of ​​the eyes over time can be created as a report 282 and provided to the requester Y.

[0321] The report 282 may output, for example, the number of times the eyes are closed per a certain period of time, the duration of eye closure (how many times the eyes are closed per minute), and other data corresponding to the requested content.

[0322] This information processing system can quantitatively compensate for extremely small changes in the image location, automatically process long-duration dynamic images, and improve overall processing capacity.

[0323] (Strong security)

[0324] Reference Figure 29 The security of this information processing system is explained.

[0325] Figure 29 This diagram shows an example of a configuration for forming a dedicated network connection with strong security.

[0326] This information processing system can be configured with a security level that is tailored to the user's policy and cost. Furthermore, communication encryption is mandatory, and is based on https.

[0327] Examples of system configurations based on security levels include "Plum," "Bamboo," and "Matsu." "Plum" is accessible only through the GIP of an in-house proxy. "Bamboo" is accessible via a reverse proxy using client certificates. "Matsu" connects to the in-house environment through a dedicated network connection (DX).

[0328] An example of the configuration of the above-mentioned "loose" dedicated network connection is shown in Figure 29 .

[0329] (Deepening of this information processing system)

[0330] Reference Figure 30 The following describes the details of this information processing system.

[0331] Figure 30 This is a diagram showing the control panel screen of this information processing system.

[0332] like Figure 30 As shown, the present information processing system has a function of displaying a control panel screen 500 that facilitates information management.

[0333] The control panel screen 500 has the following features: it can be expanded according to the animal species and experimental environment, can be output in a form other than the experimental report, and can use artificial intelligence to discover or notify feature points that humans are not aware of.

[0334] Specifically, in the control panel screen 500, various materials required for conducting experiments, such as a feed storage facility installation approval application form, an animal experiment plan, an animal experiment result report, and self-inspection or evaluation items, can be created.

[0335] By customizing the system to meet your specific business needs, you can expand the range of applicable animals and experimental content. Output can be made in formats other than experimental reports.

[0336] Furthermore, in the dashboard screen 500 , “discoveries” can be expanded through machine learning deepening. For example, “discoveries” within the scope of human cognition or “discoveries” beyond the scope of human cognition are possible.

[0337] Machine learning and function deepening can expand the scope of applicable animals and experimental content. AI can be used to discover or inform features that humans are unaware of.

[0338] In summary, in the information processing system of the third embodiment, when the analysis attributes of the image of the analysis object (the type of the analysis object (e.g., mouse, rat, etc.), the shooting direction of the analysis object (e.g., upward, obliquely upward, horizontal, etc.), the color of the analysis object (e.g., white, black, etc.)) are specified, the model selection unit 472 selects a suitable skeleton estimation model from the plurality of skeleton estimation models in the model DB 42 according to the analysis attributes of the specified analysis object image. Figure 3 The part extraction unit 52, Figure 16 The skeleton estimation model of the object of the outline detection unit 57 is selected from a plurality of individual recognition models. Figure 3 、 Figure 16 The individual identification unit 54 uses the individual identification model of the object, and according to the analysis instructions of the image, uses the selected skeleton estimation model and individual identification model to analyze the image, so that the model suitable for the analysis object can be used to correctly identify the type of animal and obtain analysis results such as the animal's behavior.

[0339] In addition, it is applicable to a variety of animal species and experimental contents, and the analysis data can be reused by having functions such as management and downloading of analysis data in the server 200, which can greatly reduce the number of animals actually used in the experiment.

[0340] In addition, it also has the following effects.

[0341] For example, analysis data is automatically tagged (labeled) based on designated analysis attributes, so that when analysis results are reused, they can be searched based on the labels, making it easy to search for analysis data.

[0342] When using only a single model for multiple analysis attributes, relearning and rechecking the accuracy of the existing model are required to update it. However, by maintaining multiple models for multiple analysis attributes, new models can be created when new attributes appear, making model learning more compact.

[0343] The above-described series of processes can be executed by hardware or by software.

[0344] in other words, Figure 3 、 Figure 16 、 Figure 24 The functional configuration is merely an example and is not particularly limited.

[0345] That is, the information processing system only needs to have the function of executing the above series of processing as a whole, and the function blocks and databases used to implement the function are not particularly limited. Figure 3 、 Figure 16 、 Figure 24 In addition, the location of the function block and the database is not particularly limited to Figure 3 、 Figure 16 、 Figure 24 The functional blocks and database of the image processing device 2 and the server 200 may also be moved to the camera 1, the PC 320, etc. Furthermore, the image processing device 2, the camera 1, and the PC 320 may be the same hardware.

[0346] Furthermore, when a series of processes are executed by software, for example, a program constituting the software is installed from a network or a recording medium into a computer or the like.

[0347] The computer may be a computer built into dedicated hardware. In addition, the computer may be a computer that can execute various functions by installing various programs, for example, in addition to a server, it may also be a general-purpose smart phone or personal computer.

[0348] In addition, for example, a recording medium including such a program is constituted not only by a removable medium (not shown) configured separately from the device body to provide the program to the user, but also by a recording medium provided to the user in a state pre-installed in the device body, etc.

[0349] Furthermore, in this specification, steps describing a program recorded on a recording medium include not only processes performed in time series in their order but also processes not necessarily performed in time series but executed in parallel or individually.

[0350] In addition, in this specification, the term "system" refers to an entire device composed of a plurality of devices, a plurality of units, and the like.

[0351] In addition, in the above embodiment, although the mark is set to a triangle or a circle, these are not limited to these examples and other shapes are also possible. In addition, the same applies to objects representing the outline of the body.

[0352] In the above embodiment, the behavior determination unit 56 determines whether the conditions defining the positional relationship between an animal's body part and other specific parts (drinking area, food area, or parts of other animals) are satisfied. However, the conditions defining the positional relationship between an animal's body part and other specific parts are stored in the material DB 43 by the storage unit 18, and the behavior determination unit 56 reads these conditions from the material DB 43. Alternatively, a behavior database may be provided separately from the material DB 43, and the conditions may be stored in the behavior database.

[0353] In the above embodiment, a machine learning method is used, but other rules may be set, for example, the part farthest from the center of gravity is set as the nose.

[0354] In the above embodiment, the center of gravity is detected (extracted) as a body part, but in addition to this, for example, a specific part (eyes, nose, ears, etc.) of the eyes, nose, ears, feet, bones or joints can also be detected (extracted).

[0355] In the above embodiment, mice are used as an example of an individual identification target. However, by expanding the data in the model DB 42 and the material DB 43, rats, hamsters, guinea pigs, rabbits, and the like can also be analyzed. Furthermore, various animals such as pigs, cattle, sheep, and chickens, as well as dogs, cats, monkeys, and humans can also be analyzed.

[0356] In the above embodiment, body parts of one or more animals are extracted from each of the multiple unit images, the extracted parts are analyzed in a time series, and the individual animals of the one or more animals in each of the multiple unit images are identified based on the analysis results. However, this sequence is merely an example, and individual parts may be extracted after individual recognition (identification). Furthermore, individual recognition and part extraction may be performed simultaneously.

[0357] When extracting individual parts after individual recognition (identification), first, it is estimated which part of the image each individual exists in, and then cropping is performed, and then the individual parts are extracted.

[0358] In addition, when identifying individuals after detecting the parts of multiple individuals, the location of specific parts in the image is detected multiple times, and then based on the speculation of "how these positions change in the time direction", it is determined which set each part belongs to (in this case, the set of parts of a specific individual).

[0359] That is, it only needs to include the steps of understanding the location of the animal's parts, identifying the individual based on this, and finally understanding the social nature of the individual.

[0360] Below, refer to Figures 31 to 54 The data associated with various indicators (the movement trajectory, the distribution of the existence position of each area, the existence time of each area, the movement distance, the total movement amount, the body direction, speed, angular velocity, etc., for the initial movement period shortly after the mouse is placed in the cage and the period until it stabilizes) are explained to understand the movement range, movement amount, behavioral habits, rhythm, etc. of parts of the mouse (animal) body (eyes, nose, etc.) extracted from dynamic images (multiple unit images arranged in time sequence), the center of gravity position, and the outer contour (contour) as they change over time.

[0361] In addition, the graphs described below show a pair of graphs from the start of measurement to 1320 frames (44 seconds) of initial movement, and a graph from the start of measurement to 18030 frames (about 10 minutes).

[0362] First, refer to Figure 31 、 Figure 32 The relationship between the position of a mouse in a cage and its movement trajectory is explained.

[0363] Figure 31 This is a graph showing the position of the mouse in the cage and its movement trajectory from the start of measurement to 1320 frames (44 seconds) of initial movement. Figure 32 It will Figure 31 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0364] Figure 31 and Figure 32 The vertical axis represents the length of the cage floor in the depth direction in pixels (PIXEL), and the vertical axis represents the length of the cage floor in the width direction in pixels (PIXEL).

[0365] Figure 31 In the 1320FRAME diagram of the initial movement, the mouse avoids the center of the cage and moves to the edge, indicating that it feels uneasy.

[0366] Figure 32 In the graph from the start of measurement until 10 minutes, the number of mice moving toward the center and edges of the cage remains high. However, there are also areas where straight lines of movement and intersecting lines are visible. This suggests that mice are staying in these intersecting areas and engaging in certain behaviors (such as grooming). By using AI to learn this information, it is possible to identify mouse behavior and mental illness.

[0367] Then, refer to Figure 33 、 Figure 34The distribution of mice in each of the nine areas into which the cage floor was divided will be described.

[0368] Figure 33 、 34 The vertical axis and horizontal axis represent pixels (PIXEL). Figure 33 This is a graph showing the distribution of the mouse's presence position in each of nine areas (numbered 0 to 8) divided into the cage floor from the start of measurement to 1320 frames (44 seconds) of initial movement. Figure 34 It will Figure 33 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0369] Figure 33 、 34 The position of the mouse in the cage is plotted every second, and the coordinate information of the position of the mouse on the floor of the cage divided into 9 areas is assigned to the identification information of the area including the coordinates for management, thereby creating a Figure 33 、 34 chart.

[0370] Figure 33 In the 1320FRAME graph of the initial movement, there are many plots in the area near the edge of the cage (numbers 0, 5, 6, 8, etc.), which shows that the mouse spends more time near the edge.

[0371] Then continue to draw for about 10 minutes Figure 34 In the graph, while most of the movement is still near the edges, there is an increase in the fourth area in the center, indicating that the mouse is moving more frequently in the center of the cage. This suggests that the mouse is calmer than when it first started moving.

[0372] Then, refer to Figure 35 、 Figure 36 The presence time of the mouse in each of the above regions (0 to 8) is described.

[0373] Figure 35 The bar graph shows the time the mouse stayed in each of the nine zones in which the cage floor was divided from the start of measurement to 1320 frames (44 seconds) of initial movement. Figure 36 It will Figure 35 The chart then depicts a bar chart up to 18030 FRAME (approximately 10 minutes).

[0374] Figure 35 、 36 The vertical axis represents the number of plots (COUNTS), and the horizontal axis represents the region (REGION). Figure 33The numbers (0 to 8) of each area correspond to .

[0375] Figure 35 、 36 The number of drawings showing the presence of mice is counted for each area, i.e., a graph showing the cumulative presence of mice is generated for each area. Figure 33 、 34 When the plot (point) increases by 1, Figure 35 、 36 The bar count for the corresponding numbered area in the chart increases by 1.

[0376] From these Figures 33 to 36 , it is possible to visually (by appearance) tell which area of ​​the cage the mouse spends more (or less) time in.

[0377] Then, refer to Figure 37 、 Figure 38 The time the mouse spends near the edge and near the center of the cage is described.

[0378] Figure 37 It means that from the start of measurement to the initial movement of 1320 FRAME (44 seconds), the center of the cage ( Figure 33 Graph of the mouse presence time when the mouse is distinguished between the fourth area (shown as the fourth area) and the vicinity of the edge (the area other than the fourth area). Figure 38 It will Figure 37 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0379] Figure 37 、 38 The vertical axis represents the number of plots (COUNTS), and the horizontal axis represents the region (REGION). The region (REGION) is divided into two areas: the center (the fourth region) and the edge (the area outside the fourth region).

[0380] Figure 37 In the graph of the initial movement, it can be seen that most of the mice are near the edge (area other than the 4th area). Figure 38 As can be seen from the graph, although the mice are overwhelmingly near the edge (areas other than the 4th area), the frequency of mice near the center (the 4th area) becomes higher than during the initial movement.

[0381] Then, refer to Figure 39 、 Figure 40 The distance that a mouse moves per 1 second (30 frames) is described.

[0382] Figure 39 This is a graph showing the relationship between the period of 1320 frames from the start of measurement to the initial movement and the movement distance of the mouse per 1 second (30 frames). Figure 40 It will Figure 39 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0383] Figure 39 、 Figure 40 The vertical axis represents distance (PIXEL) and the horizontal axis represents time (FRAME).

[0384] Figure 39 、 Figure 40 The graph shows the length (PIXEL on the image) between the mouse's past position and current position every 30 frames, or every 1 second.

[0385] Right now Figure 39 、 Figure 40 The graph shows the distance the mouse moves per second. The higher the vertical axis swings, the longer the distance moved in a short period of time. The closer the vertical axis value is to 0, it means that the mouse stays at that position.

[0386] also, Figure 39 、 Figure 40 In each graph, the broken line near the center of each amplitude represents the moving average. In this way, by taking the average value over a certain width of time, the mouse's movement speed can be known.

[0387] Since these charts are graphs of each time series, for example Figure 40 As shown in the graph, mice that calmed down in the second half of the run tended to move less. Mice that continued to move at a steady pace were considered calm. On the other hand, mice that suddenly started running or stopped for unusually long periods of time were assumed to be mentally unstable.

[0388] Then, refer to Figure 41 、 Figure 42 Describe the body orientation of the mouse in the cage.

[0389] Figure 41 This is a graph showing the relationship between the time from the start of measurement to the initial movement of 1320 frames and the instantaneous body orientation of the mouse in the cage. Figure 42 It will Figure 41 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0390] Figure 39 、 Figure 40 The vertical axis represents angle (DEGREE), and the horizontal axis represents time (FRAME).

[0391] Figure 39 、 Figure 40In the case of a graph, when the body direction of the mouse is set to be straight (0 degrees on the vertical axis of the graph) at an arbitrary time, from that point on, for example, when the mouse faces right, Figure 41 、 Figure 42 The graph oscillates in the negative direction. For example, when a mouse turns backward, its body orientation changes by more than 150 degrees, causing the graph to oscillate violently.

[0392] Then, refer to Figure 43 、 Figure 44 The total movement distance (total movement amount) of the mouse in the cage will be described.

[0393] Figure 43 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the total movement distance of the mouse. Figure 44 It will Figure 43 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0394] Figure 43 、 Figure 44 The vertical axis represents distance (PIXEL) and the horizontal axis represents time (FRAME).

[0395] Figure 43 The graph is the total distance (total movement distance), which is a graph that accumulates the above movement distances. The measurement start point is set to 0, and the distance moved by the mouse is added to finally represent the movement distance from the start of measurement.

[0396] Figure 43 、 Figure 44 In the chart of the above moving speed chart ( Figure 39 ), it is difficult to understand the instantaneous state, but by comparing the graph of the total moving distance with the graph of the moving speed, the situation of staying at this moment can be known.

[0397] In this graph, you can see how much distance you ultimately moved, and you can also see at what point in the process the amount of movement suddenly increased.

[0398] That is, through these graphs, the behavior of mice can be understood in terms of both speed and movement amount.

[0399] For example, Figure 39 In the graph, the red dot at the front (the dot in the graph) swings up and down as time passes. However, when the red dot moves horizontally, it means that the mouse has not moved. Figure 39 The speed graph shows a value close to 0.

[0400] Then, refer to Figure 45 、 Figure 46 The behavior of mice turning in their cages is described.

[0401] Figure 45 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the turning behavior (angular velocity) of the mouse. Figure 46 It will Figure 45 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0402] Figure 45 、 46 The vertical axis represents distance×angle (PIXEL×RAD), and the horizontal axis represents time (FRAME).

[0403] Figure 45 、 46 The graph represents the change in body orientation, which is a graph of the angle of body orientation change in time series, and is obtained by multiplying the moving distance by the angle of body orientation change, such as 20 degrees, 30 degrees, etc.

[0404] Figure 45 、 46 In the graph, if you simply stay in place and rotate, the value will be 0. If you change the direction of your body while moving, the graph will swing up and down greatly.

[0405] from Figure 45 、 46 As the graph shows, for example, simply changing the body's orientation can be interpreted as looking around, but staying in one place and then changing the body's orientation and turning around can indicate a state of mind in which the mouse is desperately trying to check its surroundings. By comparing the mouse's behavior not only with its previous body orientation but also taking movement into account, different behaviors can be observed.

[0406] Then, refer to Figure 47 、 Figure 48 Describe the speed at which the mouse moves within the cage.

[0407] Figure 47 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the speed of mouse movement. Figure 48 It will Figure 47 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0408] Figure 47 、 Figure 48 The vertical axis represents distance / time (PIXEL / SEC), and the horizontal axis represents time (FRAME).

[0409] Figure 47 、 Figure 48 The diagram is conceptually similar to Figure 39 、 Figure 40 A chart of the moving speed chart.

[0410] Figure 39 、 Figure 40 The chart measures distance every 30 frames.

[0411] For example, there is data between 0 frames, 30 frames, and 60 frames, but Figure 39 、 Figure 40 In the chart, only 0 frames and 30 frames, 30 frames and 60 frames are compared to calculate the movement speed. Figure 47 、 Figure 48 In the graph, the speed is calculated by taking the difference between frame 0 and frame 1 and dividing it by the time, which can be said to be a more detailed speed indicator.

[0412] The speed at which time passes is different for mice and humans. One second for humans is equivalent to a few tenths of a second for mice.

[0413] Figure 39 、 Figure 40 The graph outputs the speed per 1 second, but this is for human perception. Figure 47 、 Figure 48 In the graph, the mice's behavior was better represented by using a more subtle sensory output speed.

[0414] Then, refer to Figure 49 、 Figure 50 Describe the angular velocity of a mouse rotating in its cage.

[0415] Figure 49 This is a graph showing the relationship between the time from the start of measurement to 1320 frames (44 seconds) of initial movement and the angular velocity of mouse movement. Figure 50 It will Figure 49 The chart then depicts the chart at 18030FRAME (about 10 minutes).

[0416] Figure 49 、 50 The vertical axis represents angle / time (DEGREE / SEC), and the horizontal axis represents time (FRAME).

[0417] Figure 45 、 Figure 46 The graph shows the relationship between the mouse's body orientation and distance, and Figure 49 、 Figure 50 The chart is Figure 45 、 Figure 46 Calculated by dividing the chart value by the time.

[0418] Figure 45 、 Figure 46In the graph, the changes in the mouse body angle are numerically expressed, but Figure 49 、 50 The graph is obtained by differentiating the change in angle with time. It shows the relationship between angle and angular velocity, just like the relationship between speed and moving distance (movement amount). Figure 45 、 Figure 46 For example, you can compare the graphs together to verify whether the mouse stops for a moment. Figure 45 、 Figure 46 In the chart, you can see both the movement speed and the movement amount.

[0419] For example, if there is a momentary bounce value, the higher the bounce height, the faster the mouse turns and moves. On the other hand, the lower the value (the closer the value is to 0), the slower the mouse turns and moves.

[0420] Next, refer to Figures 51 to 54 The following describes operations including detecting the outline and body orientation of a mouse from a moving image.

[0421] Figure 51 This is a diagram showing a unit image at a certain moment in a moving image (original moving image) obtained by photographing the inside of a cage with a camera. Figure 52 This is a diagram showing the outlines (contours) of each of a plurality of (two) mice recognized from a unit image. Figure 53 This diagram shows the orientation (the direction from the center of the body to the tip of the nose) of one of the two mice. Figure 54 This diagram shows the orientation of the other mouse (the direction from the center of the body to the tip of the nose).

[0422] In the first and second embodiments described above, examples were described in which the outline (contour) detection process and the body part extraction process were performed separately. However, by combining these processes, the behavior of the mouse can be analyzed in more detail.

[0423] In this case, at a certain moment in the original dynamic image, the extracted Figure 51 The unit image shown in FIG is extracted, and image analysis is performed on the extracted unit image, and individual recognition is performed, as shown in FIG. Figure 52 As shown, the outlines (contours) of the respective multiple (two) white mice are divided into different colors such as red and blue and output.

[0424] In addition, if Figure 53 As shown, one of the two mice ( Figure 52 The center of gravity of the mouse (the upper outer contour (red)) is detected, and the orientation of the body is detected based on the positional relationship between the ears, eyes, and nose tip. The orientation of the mouse's face (the direction of the nose tip) is detected by drawing a line segment from the center of gravity to the nose tip.

[0425] Likewise, Figure 54 As shown, the other mouse ( Figure 52 The center of gravity of the mouse (with the lower outer contour (blue)) is detected, and the orientation of the body is detected based on the positional relationship between the ears, eyes, and nose tip. The orientation of the mouse's face (the direction of the nose tip) is detected by drawing a line segment from the center of gravity to the nose tip.

[0426] In this way, by combining the processing of outline recognition and part extraction, the orientation of the mouse's face (the direction of the nose tip) can be detected, and the mouse's behavior such as turning, moving forward, and moving backward while staying in place can be analyzed in more detail.

[0427] In other words, the information processing device to which the present invention is applied can adopt various embodiments having the following configurations. The information processing device corresponds to the one having the configuration described in the second embodiment. Figure 16 The image processing device 2 having the functional configuration of the third embodiment and the Figure 24 Server 200 having the functions of

[0428] That is, the second information processing device (for example Figure 16 Image processing device 2, Figure 24 The server 200, etc.) includes:

[0429] Image acquisition unit (eg Figure 16 an image acquisition unit 51 for acquiring an image of an analysis object (e.g., a moving image or other video), wherein the image of the analysis object is obtained by photographing one or more animals moving within a certain range of motion and is composed of a plurality of unit images arranged in a time direction;

[0430] Outline detection unit (e.g. Figure 16 an outline detection unit 57) for each of the plurality of unit images, detecting the body outline (body contour, etc.) of each of the one or more animals using a skeleton estimation model, wherein the skeleton estimation model estimates and outputs the body skeleton of the animal when the unit image is input;

[0431] Individual identification unit (e.g. Figure 16 The individual recognition unit 54) is configured to detect the individual according to the outline detected by the aforementioned outline detection unit (e.g. Figure 16 The outer contour detection unit 57) inputs the time series of the outer contours (body outlines, etc.) of each of the one or more animals detected from each of the plurality of unit images into an individual recognition model to identify each individual of the one or more animals in each of the plurality of unit images, and the individual recognition model outputs the individual of the animal when the time series of the one or more outer contours of the animal's body is input;

[0432] Specify the unit (e.g. Figure 24 471 ), for specifying analysis attributes of the image of the analysis target (type of the analysis target (e.g., mouse, rat, etc.), shooting direction of the analysis target animal (e.g., upward, obliquely upward, horizontal, etc.), color of the analysis target animal (e.g., white, black, etc.)); and

[0433] Model selection unit (e.g. Figure 24 The model selection unit 472, etc.) selects a model suitable for the aforementioned outline detection unit (for example, Figure 16 and selects an object suitable for the aforementioned individual recognition unit (e.g., the outer contour detection unit 57) from the plurality of aforementioned individual recognition models. Figure 16 The object of the individual identification unit 54).

[0434] In the information processing device constructed in this manner, a plurality of skeleton estimation models and individual recognition models are prepared in advance. When the analysis attribute of the image of the analysis object is specified, the skeleton estimation model suitable for the outline detection unit (for example, Figure 16 and selects an object suitable for the individual recognition unit (e.g., the outer contour detection unit 57) from a plurality of individual recognition models. Figure 16 The object of the individual identification unit 54).

[0435] Then, when there is an indication for image analysis, the skeleton estimation model and individual identification model of the selected animal are used to analyze the multiple contours extracted from each of the multiple unit images of the image in a time series, and the individual individuals of one or more animals are identified based on the analysis results. Therefore, the individual mice can be correctly identified from images of one or more animals (such as mice, etc.) moving within a certain range of motion.

[0436] In addition, the second information processing device (eg Figure 16 Image processing device 2, etc. Figure 24 The server 200, etc.) also includes:

[0437] Generate units (e.g. Figure 16 a data frame generating unit 53) to generate a data frame indicating the probability that the aforementioned outline, which changes with the aforementioned animal's behavior, is a specific behavior; and

[0438] Behavior determination unit (e.g. Figure 16 The behavior determination unit 56) determines that the behavior corresponds to the satisfied condition when the probability value of the specific behavior in the data frame satisfies any one of one or more pre-set conditions.

[0439] With this configuration, when the probability value of a specific behavior accompanying a change in the mouse profile satisfies a condition, the behavior corresponding to the condition is determined, thereby understanding the reason for the behavior of each mouse acting within a certain behavior range.

[0440] The aforementioned behavior determination unit (eg Figure 16 The behavior determination unit 56) further includes a behavior detection unit (e.g. Figure 16 A behavior detection unit 81) detects behaviors (scratching, grooming, etc.) associated with the positional relationship when conditions specifying the positional relationship between the outline of the aforementioned animal and other parts (outlines of other animals, corners of the cage) are met.

[0441] This allows us to understand why animals are behaving within a limited range of motion, such as when scratching because they are stressed. Furthermore, grooming other animals can be understood as an attempt to build relationships with them.

[0442] The aforementioned behavior includes at least one of a behavior related to the sociality of each of the aforementioned animals, a behavior related to the interaction between animals existing in the same action range, and a behavior related to the relationship between animals.

[0443] This makes it possible to determine whether the reason for an animal's behavior is related to sociality, interaction between animals in the same range of action, or relationship between animals.

[0444] Including a marker image generation unit (eg Figure 16 The labeled image generating unit 55 is used to generate a labeled image, which is the image to be detected by the aforementioned outline detection unit (e.g. Figure 16 The body outline of each of the aforementioned animals detected by the outline detection unit 57) is consistent with the body outline indicated by the aforementioned individual recognition unit (e.g. Figure 16 The individual identification unit 54) identifies the individual mark corresponding to the above-mentioned animal.

[0445] Thus, by displaying an image with a marker, one or more animals present in the image can be identified using the marker. Furthermore, after identifying the animals, the outlines of these individuals can be detected, and their changes (body movements) can be given meaning and evaluated.

[0446] The second information processing device (for example Figure 16 Image processing device 2, etc. Figure 24 The server 200, etc.) includes:

[0447] Image acquisition unit (eg Figure 16an image acquisition unit 51) for acquiring a plurality of image groups (dynamic images), wherein the plurality of image groups are frame images composed of a plurality of unit images (pixels) obtained by photographing one or more animals moving within a certain range of motion, arranged in a time direction;

[0448] Judgment unit ( Figure 16 Individual identification unit 91, etc.), for determining to which region each of the above-mentioned unit images belongs to the above-mentioned one or more animals;

[0449] Specific units (such as Figure 16 a data frame generating unit 53) for each region of the animal, determining the position of the outline (body contour, etc.) of the animal using coordinates representing the distance from a predetermined reference point in the image; and

[0450] Individual identification unit (e.g. Figure 16 The individual recognition unit 54 analyzes the changes in the position and range of the outline determined by the specific unit over time, and recognizes (classifies) the one or more animals included in the image based on the analysis results.

[0451] In addition, a behavior determination unit (eg Figure 16 The behavior determination unit 56) is used for the individual identification unit (eg Figure 16 For each animal identified (classified) by the individual identification unit 54, it is determined whether the change in the aforementioned outline meets the pre-set conditions, and the behavior label of the aforementioned animal corresponding to the aforementioned conditions (scratching behavior, sleeping, grooming behavior, etc.) is assigned to the moment information (timestamp) of the aforementioned image including the part.

[0452] This allows you to determine whether the animal's behavior is scratching, sleeping, grooming, or the like.

[0453] (Explanation of Symbols)

[0454] 1…Camera, 2…Information processing device, 11…CPU, 41…Image DB, 42…Model DB, 43…Material DB, 51…Image acquisition unit, 52…Part extraction unit, 53…Data frame generation unit, 54…Individual recognition unit, 55…Marked image generation unit, 56…Behavior determination unit, 57…Outline detection unit, 61…Dynamic image acquisition unit, 62…Unit image generation unit, 81…Behavior detection unit, 82…Behavior prediction unit, 71, 91…Individual recognition unit, 72…Part detection unit, 92…Outline identification unit, 450…Image analysis unit, 451…Network service unit, 452…Processing unit, 461…Authentication unit, 462…Retrieval unit, 463…Retrieval result output unit, 464…Analysis data addition unit, 465…Analysis result display control unit, 471…Specification unit, 472…Model selection unit, 491…Uploaded data management unit, 492…Unprocessed data management unit

Claims

1. An information processing device, comprising: An image acquisition unit, configured to acquire an image of an analysis object, wherein the image of the analysis object is obtained by photographing one or more animals moving within a certain range of motion and is composed of a plurality of unit images arranged in a time direction; an outline detection unit, for each of the plurality of unit images, detecting a body outline of each of the at least one animal using a skeleton estimation model, wherein the skeleton estimation model estimates and outputs a body skeleton of the animal when a unit image is input; an individual recognition unit for recognizing each individual of the one or more animals in each of the plurality of unit images based on an output obtained by inputting a time series of the outer contours of the respective bodies of the one or more animals detected by the outer contour detection unit from each of the plurality of unit images into an individual recognition model, the individual recognition model outputting the individual of the animal when inputting a time series of one or more outer contours of the animal's body; a specifying unit, configured to specify an analysis attribute of the image of the analysis object; a model selection unit that selects an object suitable for the outline detection unit from a plurality of the skeleton estimation models and selects an object suitable for the individual recognition unit from a plurality of the individual recognition models based on the analysis attribute of the image specified by the specifying unit; a generating unit for generating a data frame representing a change in position of the outline that changes with the behavior of the animal; as well as The behavior determination unit determines a behavior corresponding to the satisfied condition when the position change of the outline in the data frame satisfies any one of one or more preset conditions.

2. The information processing device according to claim 1, wherein The behavior determination unit further includes a behavior detection unit that detects a behavior associated with the positional relationship when a condition defining the positional relationship between the outline of the animal and other specific parts is satisfied.

3. The information processing device according to claim 2, wherein: The behavior includes at least one of a behavior related to the sociality of each of the animals, a behavior related to the interaction between animals existing in the same action range, and a behavior related to the relationship between animals.

4. An information processing method, performed by an information processing device, comprising the following steps: An image acquisition step for acquiring images, wherein the images are obtained by photographing one or more animals moving within a certain range of motion and are composed of a plurality of unit images arranged in a time direction; An outline detection step, for each of the plurality of unit images, using a skeleton estimation model to detect the body outline of each of the one or more animals, wherein the skeleton estimation model estimates and outputs the body skeleton of the animal when the unit image is input; an individual recognition step of recognizing each individual of the one or more animals in each of the plurality of unit images based on an output obtained by inputting a time series of the outer contours of the bodies of the one or more animals detected from each of the plurality of unit images into an individual recognition model, the individual recognition model outputting the individual of the animal when inputting a time series of one or more outer contours of the body of the animal; a specifying step for specifying an analysis attribute of the image of the analysis object; a model selection step of selecting, based on the specified analysis attributes of the image, an object suitable for the outline detection step from a plurality of the skeleton estimation models, and an object suitable for the individual identification step from a plurality of the individual identification models; a generating step of generating a data frame representing a change in position of the outline that changes with the behavior of the animal; as well as The behavior determination step is to determine, when the position change of the outline in the data frame satisfies any one of one or more pre-set conditions, a behavior corresponding to the satisfied condition.

5. A non-transitory recording medium having a program stored thereon, the program being configured to cause a computer controlling an information processing device to execute a control process, the control process comprising the following steps: An image acquisition step for acquiring images, wherein the images are obtained by photographing one or more animals moving within a certain range of motion and are composed of a plurality of unit images arranged in a time direction; An outline detection step, for each of the plurality of unit images, using a skeleton estimation model to extract a body outline of each of the one or more animals, wherein the skeleton estimation model estimates and outputs the body skeleton of the animal when the unit image is input; an individual recognition step of recognizing each individual of the one or more animals in each of the plurality of unit images based on an output obtained by inputting a time series of the outer contours of the bodies of the one or more animals detected from each of the plurality of unit images into an individual recognition model, the individual recognition model outputting the individual of the animal when inputting a time series of one or more outer contours of the body of the animal; a specifying step for specifying an analysis attribute of the image of the analysis object; a model selection step of selecting, based on the specified analysis attributes of the image, an object suitable for the outline detection step from a plurality of the skeleton estimation models, and an object suitable for the individual identification step from a plurality of the individual identification models; a generating step of generating a data frame representing a change in position of the outline that changes with the behavior of the animal; as well as The behavior determination step is to determine, when the position change of the outline in the data frame satisfies any one of one or more pre-set conditions, a behavior corresponding to the satisfied condition.

Images