Individual identification device, individual identification method, and recording medium

Abstract

To provide a highly versatile action analysis system from video which can be applied to various individuals.SOLUTION: A behavior type analysis device (220) includes an image acquisition unit (221) that acquires an image including an individual, a detection unit (222) that detects the individual in the image, a position specifying unit (224) that specifies a position of a body part of the detected individual, an information acquisition unit (225) that acquires time-series information indicating a change in the specified position within a predetermined time, and a behavior type analysis unit (226) that analyzes a behavior type of the individual with reference to the acquired time-series information.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] The present invention relates to a behavioral analysis device, a behavioral analysis method, a behavioral analysis program, and a recording medium. [Background technology]

[0002] To develop labor-saving herd management techniques that can accommodate the expansion of dairy farming operations, there is a need for methods that can automatically monitor the behavior of each individual cow in a herd. One practical technique for monitoring each individual cow in a herd involves attaching wearable sensors such as accelerometers to each individual. One example of such a technique involves attaching collar-type sensors to cows to measure their activity levels, calculating their activity, rumination, and rest times, and using this information to detect cows that require attention, such as those in heat or suspected of illness.

[0003] However, with such technology, the larger the herd, the higher the implementation costs and the greater the labor required to manage the equipment. The costs of implementing and operating such technology are estimated to be several tens of thousands of yen per collar-type sensor to monitor the activity of cows, and the usage fee is several thousand yen per cow per year.

[0004] In contrast, technology for monitoring cows based on video analysis has the advantage of reducing costs and easing management since it is not necessary to attach sensors to the cows. Non-Patent Documents 1 and 2 are cited as examples of technology for monitoring cows based on video analysis. Non-Patent Document 1 describes how hoof disease can be detected by analyzing the cow's gait from depth camera footage. Non-Patent Document 2 also describes how changes in posture, which are a sign of delivery, can be detected from photographed images of the cow.

[0005] Furthermore, examples of technologies for identifying and tracking individuals include those described in Patent Documents 1 and 2. Patent Document 1 describes a configuration for detecting people moving within a specific area, acquiring still images of the people, determining whether they are residents or non-residents, and tracking their behavior over time. Patent Document 2 describes a configuration for detecting moving objects in video and extracting features for identifying people. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2017-224249 [Patent Document 2] Patent Publication No. 2019-169843 [Non-patent literature]

[0007] [Non-Patent Document 1] Sho Sunagawa et al., Information Processing Society of Japan Technical Report, Vol. 2017-CVIM-206, No. 2 (2017) [Non-patent document 2] Yusuke Okimoto et al., Proceedings of the 32nd National Conference of the Japanese Society for Artificial Intelligence (2018) Summary of the Invention [Problem to be solved by the invention]

[0008] However, the techniques described in Non-Patent Documents 1 and 2 are intended for monitoring a single individual, and are therefore not suitable for analyzing the behavior of multiple individual cows.

[0009] On the other hand, the technologies described in Patent Documents 1 and 2 monitor multiple individuals and describe the use of a machine-learned model for individual identification, but do not describe the collection of training data for machine learning. In order to make the individual identification technology versatile and applicable to various herds of cattle with different individual compositions and imaging conditions, an individual identification system using a trained model that can be easily constructed for each herd is required.

[0010] One aspect of the present invention has been made to solve the above-mentioned problems, and its purpose is to realize a highly versatile behavior analysis system from video that can be applied to a variety of individuals. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention includes the following aspects. 1) A behavioral analysis device includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the image, a position identification unit that identifies the position of a body part of the detected individual, an information acquisition unit that acquires time-series information that represents a change in the identified position within a predetermined period of time, and a behavioral analysis unit that analyzes the behavioral pattern of the individual by referring to the acquired time-series information. 2) In the behavioral analysis device described in 1), the behavioral pattern of the individual analyzed by the behavioral analysis unit is selected from the group consisting of recumbent resting, recumbent ruminating, standing resting, standing ruminating, and feeding. 3) In the behavioral analysis device described in 1) or 2), the position identification unit uses an image including an individual as input data, and inputs the image acquired by the image acquisition unit as input data into a position identification model that has been machine-trained to identify the position of a body part of the individual in the image, thereby identifying the position that is output. 4) In the behavioral analysis device according to any one of 1) to 3), the position identification unit acquires coordinates of the position of a body part of the individual. 5) In the behavioral analysis device described in any one of 1) to 4), the behavioral analysis unit inputs time-series information acquired by the information acquisition unit as input data into an analysis model that has been machine-learned to estimate a behavioral type corresponding to the time-series information, and acquires, as an analysis result, a behavioral type that is output. 6) In the behavioral analysis device described in any one of 1) to 5), the position identification unit identifies the position by referring to the posture of the individual output by inputting image data of the individual as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture. 7) The behavioral analysis device according to any one of 1) to 6) further comprises an identification unit that identifies the individual detected by the detection unit. 8) The behavioral analysis method includes an image acquisition step of acquiring an image including an individual, a detection step of detecting the individual in the image, a position identification step of identifying a position of a body part of the detected individual, an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined period of time, and a behavioral analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time-series information. 9) The behavioral analysis program is a behavioral analysis program for causing a computer to function as the behavioral analysis device described in any one of 1) to 7), and causes the computer to function as the image acquisition unit, the detection unit, the position identification unit, the information acquisition unit, and the behavioral analysis unit. 10) A computer-readable recording medium has the behavioral analysis program described in 9) recorded on it. [Effects of the Invention]

[0012] According to one aspect of the present invention, a highly versatile video behavior analysis system that can be applied to a variety of individuals can be realized. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a block diagram showing a configuration of a main part of a learning data generation device according to an aspect of the present invention; [Figure 2] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 3] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 4] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 5] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 6] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 7] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 8] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 9] 10 is a flowchart showing a learning data generation process performed by the learning data generation device. [Figure 10] 1 is a block diagram showing a configuration of a main part of a learning device according to an aspect of the present invention; [Figure 11] 10 is a flowchart showing a learning process performed by the learning device. [Figure 12] 1 is a block diagram showing a configuration of a main part of an individual identification device according to an embodiment of the present invention; [Figure 13] 10 is a flowchart showing an individual identification process performed by the individual identification device. [Figure 14] 1 is a block diagram showing a configuration of a main part of a behavior analysis device according to an aspect of the present invention. [Figure 15] 10 is a flowchart showing a behavior analysis process performed by the behavior analysis device. [Figure 16] 1 is a diagram illustrating an example of individual identification used in a learning data generation device according to an embodiment of the present invention. FIG. [Figure 17]FIG. 1 is a diagram illustrating an example of an object detection machine learning model used to detect individuals in an image in a learning data generation device according to an aspect of the present invention. [Figure 18] FIG. 1 is a diagram illustrating an example of a skeleton detection machine learning model used to detect individuals in an image in a learning data generation device according to one embodiment of the present invention. [Figure 19] 10A and 10B are diagrams illustrating an example of a form in which a body region is identified in the learning data generation device according to an aspect of the present invention. [Figure 20] 1 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from an image showing a part of the body in a learning data generation device according to an aspect of the present invention. FIG. [Figure 21] 1 is a block diagram showing a configuration of a main part of a behavioral analysis system according to an embodiment of the present invention. [Figure 22] FIG. 10 is a diagram illustrating examples of body parts of an individual identified by the behavioral analysis device. [Figure 23] FIG. 2 is a diagram illustrating an example of an individual's behavioral pattern analyzed by the behavioral analysis device. [Figure 24] 10 is a flowchart showing a behavioral analysis process performed by the behavioral analysis device. DETAILED DESCRIPTION OF THE INVENTION

[0014] [Learning data generation device] A learning data generation device according to one aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a moving object detection unit that detects a moving object in the acquired image, a framing unit that frames a body region of the detected moving object that includes the entire body of the individual, an extraction unit that extracts a body region image of the framed body region, and a generation unit that generates learning data for an individual identification model that identifies individuals from the extracted body region image.

[0015] 1 is a block diagram showing the main components of a training data generation device according to one embodiment of the present invention. The training data generation device 10 includes an image acquisition unit 11, a moving object detection unit 12, a framing unit 13, an extraction unit 14, a training data generation unit (generation unit) 15, and a training data storage unit 16. The training data generation device 10 generates training data used to build an individual identification model by machine learning.

[0016] (individual) The individual identification model, which is machine-learned using training data generated by the training data generation device 10, is a machine-learned model that identifies individuals in images of multiple individuals. The individuals identified by the individual identification model may be humans, animals, other moving objects, etc., but are preferably individuals managed in groups within a closed space. The individuals identified by the individual identification model are preferably livestock individuals raised in large numbers, even more preferably cattle, even more preferably dairy cows, and most preferably Holstein dairy cows. While the following describes Holstein dairy cows as an example of individuals to be identified, the present invention is not limited thereto. Other dairy cows or other animals can be identified in the same manner as Holstein dairy cows by utilizing differences in patterns, body shapes, or simple markers. For example, the present invention can also be applied to koi carp, pigs, cats, dogs, etc. An example of a configuration using simple markers is a configuration in which a label such as a number is attached to a collar, as shown in FIG. 16. FIG. 16 is a diagram illustrating an example of individual identification used in a training data generation device according to one embodiment of the present invention. By using such simple markers, it is possible to easily identify individuals even of species that are difficult to identify by pattern, and the present invention can be applied to individuals of various species.

[0017] The multiple animals may be free-roaming indoors as shown in FIG. 2, or may be tethered and kept on a chain or the like. FIG. 2 is a diagram illustrating the training data generation process of a training data generation device according to one aspect of the present invention. The multiple animals may also be kept in a controlled outdoor area. In this embodiment, the case where the animals are cows will be described as an example.

[0018] (Image acquisition unit 11) The image acquisition unit 11 acquires images including individual cattle. The image acquisition unit 11 acquires images including the individual cattle from the camera 20 that captured the image of the individual cattle via an input / output unit (not shown). The image acquisition unit 11 acquires images including the individual cattle from the camera 20 by wireless communication or wired communication. The image acquisition unit 11 may acquire an image every time the camera 20 captures an image of the individual cattle, or may acquire an image at predetermined time intervals.

[0019] The camera 20 may photograph the individual cattle from above or from diagonally above. In the learning data generation device 10, not only images of the individual cattle photographed from above but also images photographed from diagonally above can be suitably used. As an example, when the individual cattle are raised indoors, the camera 20 may be installed on the ceiling or on a wall. Furthermore, when the individual cattle are raised outdoors, the camera 20 may be installed by fixing it to a pole, fence, or the like.

[0020] The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken with a thermal camera or the like, or a three-dimensional image obtained with a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken with a laser sensor, it is possible to obtain images at night without using lighting. Furthermore, the image including the individual cattle may be a still image or a video. The image acquisition unit 11 preferably acquires images including the individual cattle from multiple cameras 20, and more preferably, the multiple cameras 20 acquire images of the individual cattle from different directions. The image acquisition unit 11 sends the acquired image including the individual cattle to the moving object detection unit 12.

[0021] (Motion detection unit 12) The moving object detection unit 12 detects moving objects in the acquired image including the individual cow. The moving object detection unit 12 detects the movement of objects in the image including the individual cow and detects the moving object as a moving object. The moving object detection unit 12 detects moving objects in the image by, for example, inter-frame difference processing that compares multiple image frames captured at different times, or background difference processing that compares the captured image with a background image captured in advance. The moving object detection unit 12 acquires position information of the detected moving object in the image and sends it to the enclosing unit 13.

[0022] (Boxed section 13) The framing unit 13 frames a body region including the entire body of a bovine individual among the detected moving objects. When the detected moving object is a bovine individual, the framing unit 13 frames the body region of the bovine individual. The framing unit 13 identifies the body region of the bovine individual in an image of a partial region corresponding to the acquired position information of the moving object and frames the body region.

[0023] If a frame is added to an image of a partial region corresponding to the position information of a moving object without specifying the body region of the bovine individual, the frame may be added to only a part of the body, such as the head or tail of the bovine individual, as shown in Figure 3. In addition, the frame may be added to a moving object other than the bovine individual.

[0024] On the other hand, the framing unit 13 identifies and frames the body region of the individual cow in the image of the partial region corresponding to the position information of the moving object, so that the entire body of the individual cow is framed so as to be inside the frame, as shown in Fig. 4. Fig. 4 is a diagram illustrating the learning data generation process of the learning data generation device according to one embodiment of the present invention.

[0025] The body region of the bovine individual in the framing unit 13 can be identified by, for example, a method using a trained model trained by machine learning to identify the body region of the bovine individual, or by methods such as feature point matching and pattern matching. That is, the framing unit 13 applies a frame to the body region output by inputting image data of the detected moving object as input data into a discrimination model trained by machine learning to extract a body region including the entire body of the bovine individual from an image including the bovine individual. Such a machine-learned discrimination model can be obtained by applying a known learning algorithm, such as a support vector machine, a random forest, a k-means algorithm, or a convolutional neural network, to training data such as that shown in FIG. 5 , which includes accumulated images of the entire bodies of thousands to tens of thousands of bovine individuals. FIG. 5 is a diagram illustrating the training data generation process of a training data generation device according to one embodiment of the present invention.

[0026] Furthermore, in cases where only a part of the individual cow is included in the captured image, where the individual cows are overlapping, or where a part of the individual cow is hidden by an obstacle such as a pillar or a wall, the framing unit 13 generates a body region image from an image representing a part of the individual cow and adds a frame to the generated body region image. That is, the framing unit 13 takes the image representing the entire body of the individual as a body region, which is output by inputting image data of the detected moving object as input data into a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a part of the body of the individual, and adds a frame to the body region.

[0027] Such a generative model can be a known generative model such as a variational autoencoder (VAE) or a generative adversarial network (GAN). For example, when a variational autoencoder is used, as shown in FIG. 6, a numerical value representing the features of image data is obtained by convolution of an encoder from image data of a portion of a body of a bovine individual. Then, an image of the entire body of the bovine individual is reconstructed by deconvoluting the numerical value representing the features. FIG. 6 is a diagram illustrating the training data generation process of a training data generation device according to one embodiment of the present invention. For example, as shown in FIG. 6, a machine-learned generative model can be obtained by applying a known generative model to training data accumulated with images of the entire bodies of thousands to tens of thousands of bovine individuals. In this way, by using a generative model trained to reconstruct a body region image of the entire body from an image of a portion of the body of a bovine individual, an image of the entire body of the bovine individual can be reconstructed using an image of a portion of the body of the bovine individual as input image data. Note that in the example shown in FIG. 6, the data output from the generative model may be only a frame surrounding the entire body, and patterns, colors, etc. may not be output.

[0028] 7 and 8 are diagrams illustrating the training data generation process of a training data generation device according to one embodiment of the present invention. When image data representing a portion of a body of a bovine individual is input to a machine-learned generative model, as shown in FIG. 7, a numerical value Z representing the characteristics of the image data is obtained by convolution in the decoder. The numerical value Z is a value representing the characteristics of the input image, such as information about the scale of the image or the extent of the entire body to which the area included in the image corresponds. Then, by deconvolving the obtained numerical value Z, a body region image representing a body region of the bovine individual is generated and output. Furthermore, when image data representing a portion of a body of a bovine individual that includes features not included in the training data is input to this machine-learned generative model, as shown in FIG. 7, a body region image of the bovine individual is generated and output. In this way, even when unknown bovine image data is input to the generative model, data framing the entire body of the bovine individual is output. In this way, even when image data representing a portion of a body of a bovine individual that includes features not included in the training data is input to the generative model, data framing the entire body of the bovine individual is output. Furthermore, even when the scale of an image representing a part of the body of a bovine individual is larger than that of Figure 7, as in the input data shown in Figure 8, an image of the body region of the bovine individual is output, surrounded by a frame of a size according to the scale of the input image.

[0029] In this way, when the image of the moving object is only a part of the cow individual, the framing unit 13 generates the body area of ​​the cow individual and frames it by using a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a part of the body of the individual.

[0030] (Extraction part 14) The extraction unit 14 extracts the body region image with the frame added. The extraction unit 14 removes the image outside the frame from the image captured by the camera 20 and cuts out the body region image inside the frame. The extraction unit 14 sends the extracted body region image to the learning data generation unit 15.

[0031] (Learning data generation unit 15) The learning data generation unit 15 generates learning data for an individual identification model that identifies individuals from the extracted body region images. The learning data may be unlabeled unsupervised data, and may include, for example, hundreds to tens of thousands of body region image data. The learning data generation unit 15 may generate learning data by classifying and accumulating the body region images for each individual. The learning data generation unit 15 may generate learning data from the accumulated body region images when a predetermined number of body region images have been accumulated or when an instruction to generate learning data is received from a user.

[0032] The classification of body region images for each individual by the learning data generation unit 15 can be performed by, for example, referring to information such as the similarity of the body region images, the positional relationship with previous and next images in a time series, etc. Such classification of body region images for each individual may be performed using a classification estimation model that is machine-learned to estimate the body region image based on the similarity of the body region image or the positional relationship with previous and next images in a time series.

[0033] The training data generation unit 15 generates training data corresponding to the population identified by the individual identification model. For example, when the individual identification model that performs machine learning using the training data identifies population X including 10 bovine individuals from bovine individual No. 1 to bovine individual No. 10, the training data generation unit 15 generates training data including body region image data of the 10 bovine individuals included in population X. The training data generation unit 15 stores the generated training data in the training data storage unit 16 in association with information on the corresponding population.

[0034] (Learning data storage unit 16) The generated learning data is stored in the learning data storage unit 16. The learning data storage unit 16 may also store data such as images acquired by the image acquisition unit 11 and position information in the images of moving objects detected by the moving object detection unit 12. The learning data storage unit 16 may also store a program indicating a series of processes required for the operation performed by the learning data generation device 10.

[0035] (Learning data generation process) An example of the learning data generation process performed by the learning data generation device 10 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the learning data generation process performed by the learning data generation device 10 of Fig. 1.

[0036] First, in step S91, the image acquisition unit 11 acquires an image including an individual cow. Note that the learning data generation process by the learning data generation device 10 may be started by accepting input of captured image data from the camera 20.

[0037] Next, the moving object detection unit 12 detects a moving object in the acquired image including the individual cow (step S92). Subsequently, in step S93, the framing unit 13 frames the body region of the individual cow among the detected moving objects. Next, in step S94, the extraction unit 14 extracts the framed body region image. Then, the learning data generation unit 15 generates learning data from the body region image (step S94). The learning data generation unit 15 stores the generated learning data in the learning data storage unit 16, and ends the learning data generation process.

[0038] In this way, the training data generation device 10 automatically generates training data used to build an individual identification model through machine learning by extracting and storing body region images of bovine individuals for which motion detection has been performed. This makes it easy to generate training data for building an individual identification model. Therefore, training data can be easily generated for each population of various individuals with different individual configurations and imaging conditions. Furthermore, the training data generated by the training data generation device 10 includes body region images of bovine individuals in various orientations. Therefore, by identifying individuals using an individual identification model trained using such training data, individual identification becomes possible regardless of the orientation of the bovine individuals in the image to be identified.

[0039] In machine learning for image recognition, the time and effort required to prepare training data has traditionally been a problem. The training data generation device 10 automatically accumulates training data used in machine learning for individual identification models. This makes it possible to easily build individual identification models that are compatible with each dairy farm, where the individual composition of cattle populations and photography conditions vary, thereby enabling the general-purpose use of individual identification models.

[0040] [Learning device] A learning device according to one embodiment of the present invention includes a learning data acquisition unit that acquires learning data generated by a learning data generation device according to one embodiment of the present invention, and a learning unit that constructs an individual identification model that identifies individuals by using the acquired learning data to perform machine learning to identify individuals contained in an image.

[0041] 10 is a block diagram showing the configuration of the main parts of a learning device according to one embodiment of the present invention. The learning device 100 includes a learning data acquisition unit 101, a learning unit 102, and a learning model storage unit 103. The learning device 100 constructs an individual identification model that identifies individuals.

[0042] (Learning data acquisition unit 101) The training data acquisition unit 101 acquires training data stored in the training data storage unit 16 of the training data generation device 10 via an input / output unit (not shown). The training data acquisition unit 101 acquires training data corresponding to a population to be identified by an individual identification model that is machine-learned in the training unit 102. For example, if the individual identification model is a trained model that identifies cattle individuals of population X that includes 10 cattle individuals from cattle individual No. 1 to cattle individual No. 10, the training data acquisition unit 101 acquires training data corresponding to population X. The training data acquisition unit 101 sends the acquired training data to the training unit 102.

[0043] (Learning Section 102) The learning unit 102 uses the acquired learning data to construct an individual identification model that identifies individuals. The learning unit 102 uses the acquired learning data to perform machine learning of the individual identification model by applying a known learning algorithm such as a support vector machine, a random forest, a k-means method, or a convolutional neural network.

[0044] The machine-learned individual identification model constructed in the learning unit 102 is a trained model that outputs identification information for identifying an individual when image data including the individual is input as input data. For example, when the individual identification model identifies a cow individual in population X, the individual identification model outputs which cow individual in population X (cow individual No. 1, cow individual No. 2, ..., or cow individual No. 10) the cow individual in the input image data corresponds to. The learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103.

[0045] (Learning model storage unit 103) The learning model storage unit 103 stores an individual identification model that has undergone machine learning. The learning model storage unit 103 may also store learning data acquired by the learning data acquisition unit 101. The learning model storage unit 103 may also store a program that indicates a series of processes required for the operation of the learning device 100.

[0046] (Learning process) An example of the learning process performed by the learning device 100 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the learning process performed by the learning device 100 of Fig. 10.

[0047] First, in step S111, the training data acquisition unit 101 acquires training data from the training data generation device 10. Next, the training unit 102 uses the acquired training data to train an individual identification model that identifies individuals (step S112). Then, the training unit 102 stores the trained individual identification model in the training model storage unit 103 (step S113), and ends the training process.

[0048] In this way, the learning device 100 constructs an individual identification model through machine learning using learning data generated for each population, making it possible to easily construct an individual identification model that corresponds to each dairy farm site, where the individual composition of the cattle population and the photographing conditions vary, for example. As a result, the individual identification model can be used for general purposes.

[0049] [Individual Identification Device] An individual identification device according to one embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, and an identification unit that outputs individual identification information of the individual included in the image by inputting data of the acquired image as input data into the individual identification model constructed by a learning device according to one embodiment of the present invention.

[0050] 12 is a block diagram showing the main configuration of an individual identification device according to one embodiment of the present invention. The individual identification device 120 includes an image acquisition unit 121, an identification unit 122, and an identification image storage unit 123. The individual identification device identifies individuals in a population using an individual identification model.

[0051] (individual) The individual identified by the individual identification device 120 may be a human, an animal, another moving object, etc., but is preferably an individual managed in groups in a closed space. The individual identified by the individual identification model is more preferably a livestock individual raised in large numbers, even more preferably a cattle individual, even more preferably a dairy cow individual, and most preferably a Holstein dairy cow individual. Note that the raised large numbers of individuals may be individuals that are allowed to roam free indoors, as shown in FIG. 2, or individuals that are kept tethered by a chain or the like. The raised large numbers of individuals may also be individuals that are raised in a managed area outdoors. In this embodiment, a case in which the individuals are cattle individuals will be described as an example.

[0052] (Image acquisition unit 121) The image acquisition unit 121 acquires images including individual cattle. The image acquisition unit 121 acquires images including the individual cattle from the camera 20 that captured the image of the individual cattle via an input / output unit (not shown). The image acquisition unit 121 acquires images including the individual cattle from the camera 20 by wireless communication or wired communication. The image acquisition unit 121 may acquire an image every time the camera 20 captures an image of the individual cattle, or may acquire an image at predetermined time intervals.

[0053] The camera 20 may photograph the individual cattle from above or from diagonally above. In the individual identification device 120, not only images of the individual cattle photographed from above but also images photographed from diagonally above can be suitably used. As an example, when the individual cattle are raised indoors, the camera 20 may be installed on the ceiling or on a wall. Furthermore, when the individual cattle are raised outdoors, the camera 20 may be installed by fixing it to a pole, fence, etc.

[0054] The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken with a thermal camera or the like, or a three-dimensional image obtained with a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken with a laser sensor, it is possible to obtain images at night without using lighting. Furthermore, the image including the individual cattle may be a still image or a video. The image obtaining unit 121 sends the obtained image including the individual cattle to the identification unit 122.

[0055] (Identification unit 122) The identification unit 122 inputs the data of the acquired image as input data into the individual identification model constructed by the learning device 100, and outputs individual identification information of the individual cattle in the image. The identification unit 122 acquires the individual identification model stored in the learning model storage unit 103 of the learning device 100 via an input / output unit (not shown). The individual identification model is a machine-learned model corresponding to the population identified by the individual identification device 120. For example, if the population identified by the individual identification device 120 is population X including 10 individual cattle, from cattle individual No. 1 to cattle individual No. 10, the individual identification model is a machine-learned model corresponding to population X.

[0056] The identification unit 122 associates individual identification information, which indicates which individual bovine animal is included in the acquired image, with the image data and stores the associated information in the identified image storage unit 123. Furthermore, the identification unit 122 may add the individual identification information to the image data and display it on a display unit (not shown), as shown in Fig. 4. For example, as shown in Fig. 4, the identification unit 122 displays the individual bovine animal in the image by adding a bovine individual number that identifies the individual bovine animal.

[0057] The image identified by the identification unit 122 may be an image obtained by extracting a body region image including the entire body of the individual bovine from the image acquisition unit 121. That is, the individual identification device 120 may further include a framing unit that adds a frame to the body region of the individual bovine in the image acquired by the image acquisition unit 121, and an extraction unit that extracts the framed body region image. The individual identification device 120 may also include a moving object detection unit that detects a moving object in the image acquired by the image acquisition unit 121. The identification unit 122 then inputs the body region image extracted by the extraction unit as input data into an individual identification model, thereby outputting individual identification information for the individual bovine in the image. Note that the image identified by the identification unit 122 may be the image itself acquired by the image acquisition unit 121 from the camera 20, for example, when the individual bovine in the image is alone.

[0058] In this way, the individual identification device 120 may use, as the image to be identified, a body region image that has been subjected to processing similar to the learning data generation processing by the learning data generation device 10, rather than the image itself acquired from the camera 20. This prevents the image to be identified from including parts unnecessary for individual identification, enabling more accurate individual identification.

[0059] (Identification image storage unit 123) Individually identified images are stored in the identification image storage unit 123. The identification image storage unit 123 may also store a program indicating a series of processes required for the operation of the individual identification device 120.

[0060] (Individual identification processing) An example of the individual identification process performed by the individual identification device 120 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the individual identification process performed by the individual identification device 120 of Fig. 12.

[0061] First, in step S131, the image acquisition unit 121 acquires an image including an individual cow. Note that the individual identification process by the individual identification device 120 may be started by receiving input of captured image data from the camera 20.

[0062] Next, the identification unit 122 reads out the trained individual identification model from the learning device 100 (step S132). Subsequently, in step S133, the identification unit 122 inputs image data into the read individual identification model. Then, in step S134, the identification unit 122 associates the individual identification information of the individual cow output by the individual identification model with the image, stores them in the identified image storage unit 123, and ends the individual identification process.

[0063] In this way, the individual identification device 120 identifies individuals using an individual identification model corresponding to each population, making it possible to identify individuals corresponding to each dairy farm site where the individual composition of the cattle population and the photographing conditions vary, for example. Furthermore, by using an individual identification model trained by the learning device 100, it is possible to identify individuals regardless of the orientation of the individual cattle in the image to be identified. The individual identification device 120 allows for general-purpose use of the individual identification model.

[0064] [Behavior analysis device] A behavioral analysis device according to one embodiment of the present invention includes an image acquisition unit that acquires images including individuals, an identification unit that outputs individual identification information for the individuals included in the images by inputting data from the acquired images as input data into an identification model for identifying individuals constructed by machine learning to identify the individuals included in the images using learning data generated by the learning data generation device, an identification image acquisition unit that acquires time-series images of the identified individuals, and an analysis unit that analyzes the behavior of the individuals by referring to the acquired images. That is, the behavioral analysis device includes an identification image acquisition unit that acquires time-series images of the individuals identified by the individual identification device according to one embodiment of the present invention, and an analysis unit that analyzes the behavior of the individuals by referring to the acquired images.

[0065] 14 is a block diagram showing the configuration of the main parts of a behavior analysis device according to one embodiment of the present invention. Behavior analysis device 140 includes identification image acquisition unit 141, analysis unit 142, and analysis result storage unit 143. The behavior analysis device tracks and analyzes the behavior of identified individuals.

[0066] (Identification image acquisition unit 141) The identification image acquisition unit 141 acquires, via an input / output unit (not shown), images of individual cattle stored in the identification image storage unit 123 of the individual identification device 120. The identification image acquisition unit 141 acquires time-series images of individual cattle identified by the individual identification device 120. The identification image acquisition unit 141 sends the acquired time-series images of individual cattle to the analysis unit 142.

[0067] (Analysis Department 142) The analysis unit 142 analyzes the behavior of the individual cattle by referring to the time-series images of the individual cattle. The time-series images of the individual cattle make it possible to track the behavior of the individual cattle. The analysis unit 142 analyzes the behavior of the individual cattle over a predetermined period of time by tracking the behavior of the individual cattle based on the time-series images of the individual cattle. The analysis unit 142 analyzes, for example, the amount of movement, range of movement, and pattern of movement of the individual cattle over a predetermined period of time by tracking the behavior of the individual cattle based on the time-series images of the individual cattle. The analysis unit 142 stores the analysis results of the individual cattle in the analysis result storage unit 143.

[0068] The analysis unit 142 may predict the physical condition, estrus state, disease, etc. of the individual cattle by analyzing behavioral patterns such as the time the individual cattle spends sitting, standing, moving around, etc. Furthermore, the analysis unit 142 may manage the feeding of the individual cattle by analyzing behavioral patterns such as the number of times and times the individual cattle drinks water, and the number of times and times the individual cattle eats feed.

[0069] The analysis unit 142 may acquire position coordinates in the image of the individual bovine animal from the image of the individual bovine animal and convert the position coordinates into spatial coordinates of the space in which the individual bovine animal exists, thereby tracking and analyzing the range of movement, behavioral patterns, etc. of the individual bovine animal within the space. The conversion from position coordinates to spatial coordinates can be performed, for example, by converting the acquired position coordinates into spatial coordinates of the space in which the individual bovine animal exists, using information on a reference point on the image whose spatial coordinates are known. If the image of the individual bovine animal referenced by the analysis unit 142 is a body region image including the entire body of the individual bovine animal, the position coordinates of the individual bovine animal's feet can be acquired, thereby enabling the behavior of the individual bovine animal within the space to be accurately tracked. The position coordinates or spatial coordinates of the individual bovine animal's feet may be displayed on a display unit (not shown) together with the body region image of the individual bovine animal, as shown in FIG. 4.

[0070] (Behavioral analysis processing) An example of behavior analysis processing by behavior analysis apparatus 140 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing behavior analysis processing by behavior analysis apparatus 140 of Fig. 14.

[0071] First, in step S151, the identification image acquisition unit 141 acquires time-series image data of the individual cow. Next, the analysis unit 142 tracks the behavior of the individual cow using the time-series images of the individual cow and analyzes the behavior of the individual cow (step S152). The analysis unit 142 stores the analysis results in the analysis result storage unit 143 and ends the behavior analysis process.

[0072] In this way, behavior analysis device 140 analyzes the behavior of individuals by tracking images of individuals identified using an individual identification model corresponding to each population, making it possible to perform accurate and easy individual behavior analysis that is suited to, for example, each dairy farm site, where the individual composition of the cattle population and the shooting conditions vary. Note that a behavior analysis system equipped with learning data generation device 10 shown in Fig. 1, learning device 100 shown in Fig. 10, individual identification device 120 shown in Fig. 12, and behavior analysis device 140 shown in Fig. 14 are also included in the scope of the present invention.

[0073] [Method for generating learning data and method for analyzing behavior] A training data generation method according to one aspect of the present invention includes an image acquisition step of acquiring an image including an individual, a moving object detection step of detecting a moving object in the acquired image, a framing step of framing a body region of the detected moving object that includes the entire body of the individual, an extraction step of extracting a body region image of the framing-added body region, and a generation step of generating training data for a discriminative model that identifies the individual from the extracted body region image. As an example, the training data generation method according to one aspect of the present invention is realized by the training data generation device according to one aspect of the present invention described above. Therefore, the description of the training data generation method according to one aspect of the present invention follows the description of the training data generation device according to one aspect of the present invention described above.

[0074] A behavioral analysis method according to one aspect of the present invention includes an image acquisition step of acquiring images including individuals; an identification step of inputting data from the acquired images as input data into an identification model for identifying individuals, the model being constructed by machine learning to identify individuals included in the images using learning data generated by the learning data generation method; an identification image acquisition step of acquiring time-series images of the identified individuals; and an analysis step of analyzing the behavior of the individuals by referring to the acquired images. As an example, the behavioral analysis method according to one aspect of the present invention is realized by the behavioral analysis device according to the above-described aspect of the present invention. Therefore, the description of the behavioral analysis method according to one aspect of the present invention follows the description of the behavioral analysis device according to the above-described aspect of the present invention.

[0075] [Software implementation example] The control blocks of the learning data generation device 10, the learning device 100, the individual identification device 120, and the behavior analysis device 140 may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0076] In the latter case, each of the above-mentioned devices includes a computer that executes instructions from a program, which is software that realizes each function. This computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. The object of the present invention is achieved when the processor in the computer reads and executes the program from the recording medium. The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transitory tangible medium," such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The device may also include a RAM (Random Access Memory) for expanding the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0077] In other words, the learning data generation device, learning device, individual identification device, and behavioral analysis device 140 according to each aspect of the present invention may be realized by a computer. In this case, the program that realizes each device on a computer by making the computer operate as each part (software element) of each device, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention.

[0078] Specifically, the following programs also fall within the scope of the present invention: A program for causing a computer to function as a learning data generation device according to each aspect of the present invention, the program comprising: an image acquisition step for acquiring an image including an individual; a moving object detection step for detecting a moving object in the acquired image; a framing step for adding a frame to a body region including the entire body of the individual among the detected moving objects; an extraction step for extracting a body region image of the body region to which the frame has been added; and a generation step for generating learning data for a discriminative model that identifies individuals from the extracted body region image: A program for causing a computer to function as a behavioral analysis device according to any one of the aspects of the present invention, the program comprising: an image acquisition step of acquiring an image including an individual; an identification step of inputting data of the acquired image as input data into an identification model for identifying an individual, the identification model being constructed by machine learning to identify the individual included in the image using learning data generated by the learning data generation device according to any one of the aspects of the present invention, thereby outputting individual identification information of the individual included in the image; an identification image acquisition step of acquiring time-series images of the identified individual; and an analysis step of analyzing the behavior of the individual by referring to the acquired images. A program for causing a computer to function as a learning device according to each aspect of the present invention, the program comprising: a learning data acquisition step for acquiring learning data generated by the learning data generation device according to each aspect of the present invention; and a learning step for constructing an identification model for identifying individuals by machine learning using the acquired learning data to identify individuals included in an image. A program for causing a computer to function as an individual identification device relating to each aspect of the present invention, characterized in that the program executes an image acquisition step of acquiring an image including an individual, and an identification step of inputting data of the acquired image as input data into the identification model constructed by the learning device relating to each aspect of the present invention, thereby outputting individual identification information of the individual included in the image.

[0079] [Modification] A learning data generation device according to another aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, a body region identification unit that identifies a body region including the entire body of the detected individual, an extraction unit that extracts a body region image of the identified body region, and a generation unit that generates learning data for a discrimination model that identifies individuals from the extracted body region image.

[0080] That is, in the training data generation device, the method for detecting individuals in an image is not limited to a moving object detection method, and may be realized by other methods for detecting individuals. Another example of detecting individuals in an image is a method using machine learning. Therefore, a configuration in the training data generation device that uses a detection unit that detects individuals in an image using machine learning instead of a moving object detection unit is also included in the scope of the present invention.

[0081] When detecting an individual in an image using machine learning, it is preferable to create a machine learning model suitable for detecting the individual depending on the direction in which the image was taken. For example, when using an image of an individual taken from diagonally above, a machine learning model generated to be suitable for detecting the individual in an image taken from diagonally above is used. Methods for detecting an individual in an image using machine learning include methods for detecting an object using machine learning and methods for detecting the skeleton of an individual. Such a detection unit can also be used in an individual identification device according to one aspect of the present invention.

[0082] When using a machine learning model created to detect objects from an image, image data of individual cows is input as input data to the object detection machine learning model, and a detection result of the individual cow part in the image is output, as shown in Figure 17. When using a machine learning model created to detect skeletons from an image, image data of individual cows is input as input data to the skeleton detection machine learning model, as shown in Figure 18. As a result, the skeleton positions of individual cows in the image are detected and organized for each individual cow, and a detection result of the individual cow part in the image is output.

[0083] Furthermore, in the training data generation device, the method for identifying the body region including the entire body of a detected individual is not limited to the method of adding a frame, and other methods for identifying the body region of an individual may be used. Another example of identifying the body region of an individual is a method using image segmentation, as shown in FIG. 19 . FIG. 19 is a diagram illustrating an example of how a body region is identified in a training data generation device according to one embodiment of the present invention. As shown in FIG. 19 , a body region is identified by image segmentation in an image including a cow individual, and, for example, position coordinates indicating only the identified body region range are obtained. Thus, a configuration in which a training data generation device uses a body region identification unit that identifies the body region of an individual by image segmentation instead of a frame is also included in the scope of the present invention. Furthermore, such a body region identification unit can also be used in an individual identification device according to one embodiment of the present invention.

[0084] Furthermore, in the learning data generation device, the method for identifying an image of the entire body from an image representing a part of the body of an individual is not limited to the method of restoring images representing other parts of the body, and may be realized by other methods for identifying an image of the entire body from an image representing a part of the body. Another example of identifying an image of the entire body from an image representing a part of the body is a method of referring to a posture estimated based on skeletal detection using machine learning.

[0085] An example of identifying an image of the entire body from an image showing a part of the body by referring to the posture estimated based on skeleton detection is shown in Fig. 20. Fig. 20 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from an image showing a part of the body in a learning data generation device according to an embodiment of the present invention. As shown in Fig. 20, when image data including only a part of an individual cow is acquired, the positions of parts not included in the image (invisible parts) are identified by referring to the posture estimated from parts included in the image (visible parts).

[0086] For example, in the image data of Figure 20, only seven parts are visible: the nose, left eye, right eye, left ear, right ear, left shoulder, and left hip. When this image data is input into a posture estimation model, the skeleton of the visible parts is detected through machine learning. Then, skeletal frame information including the positions of the 10 invisible parts (right shoulder, right hip, left front ankle, left front hoof, left hind ankle, left hind hoof, right front ankle, right front hoof, right hind ankle, and right hind hoof) is output as the posture estimation result. Note that the parts of the bovine individual shown here are examples, and the types and numbers of parts included and not included in the image are not limited to these and can be set as appropriate.

[0087] This allows the entire body region to be identified and used for behavioral analysis of the individual, even when there is an obstruction in the image and only a portion of the individual is included.In this way, information on the body parts included in the image is used as information for estimating the body parts not included in the image.In addition, the positions of each body part detected in images before and after the target image in a time series may be used as input data, for example.

[0088] In this way, the scope of the present invention also includes a configuration in which, in a learning data generation device, a body region identification unit identifies a body region representing the entire body of an individual by referring to the posture of the individual output by inputting image data of the detected moving object as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture of the individual. Furthermore, such a body region identification unit can also be used in an individual identification device according to one aspect of the present invention.

[0089] [Behavioral Analysis System] The behavioral type (behavioral pattern) of each individual animal is important information for animal management. For example, feeding behavior and resting behavior are indicators of whether maintenance behavior is being ensured. Furthermore, in ruminant livestock, rumination behavior, among other behavioral patterns, is considered useful information that serves as a barometer reflecting feed intake, signs of various diseases, recovery status, or stress conditions. Conventionally, systems that use wearable sensors to grasp behavioral patterns have been put into practical use, but they have problems in terms of cost and management effort. Meanwhile, conventional image analysis has had difficulty detecting behaviors that require movement understanding, such as rumination. According to one embodiment of the present invention, by grasping an animal's behavioral pattern using image analysis, behavioral pattern data can be obtained easily and labor-savingly. Furthermore, according to one embodiment of the present invention, even behavioral patterns that require movement understanding can be detected.

[0090] A behavioral analysis system according to one embodiment of the present invention will be described with reference to Fig. 21. Fig. 21 is a block diagram showing the configuration of the main parts of a behavioral analysis system 200 according to one embodiment of the present invention. The behavioral analysis system 200 includes a behavioral analysis device 220. The behavioral analysis system 200 also includes a camera 20, a learning data generation device 210, and a learning device 100.

[0091] (Behavioral analysis device 220) The behavioral analysis device 220 includes an image acquisition unit 221, a detection unit 222, an identification unit 223, a position identification unit 224, an information acquisition unit 225, and a behavioral analysis unit 226. The behavioral analysis device 220 further includes an analysis result storage unit 227. The behavioral analysis device 220 detects and analyzes the behavioral patterns of individuals.

[0092] <Individual> The individuals whose behavioral patterns are analyzed by the behavioral analysis device 220 may be humans, animals, other mobile objects, etc., but may also be individuals managed in groups in a closed space or managed individually. Such individuals include livestock individuals raised in large numbers and livestock individuals raised alone. Livestock individuals may be ruminant animals such as cattle, for example, dairy cows, more specifically, Holstein dairy cows. Individuals raised in large numbers may be individuals allowed to roam free indoors or individuals kept tethered by chains or the like. Additionally, individuals raised in large numbers may be individuals raised in a managed outdoor area. In this embodiment, an example will be described in which the individuals are cattle.

[0093] <Image acquisition unit 221> The image acquisition unit 221 acquires an image including an individual cow photographed by the camera 20. The image acquisition unit 221 is configured in the same manner as the image acquisition unit 121 of the individual identification device 120 described above. Therefore, the description of the image acquisition unit 121 of the individual identification device 120 will be used to explain the image acquisition unit 221, and a detailed description thereof will be omitted. The camera 20 is also the same as the camera 20 described in relation to the individual identification device 120, and a detailed description thereof will be omitted.

[0094] <Detection unit 222> The detection unit 222 detects individual cattle in the image acquired by the image acquisition unit 221. Methods for detecting individual cattle in an image include a moving object detection method and a method using machine learning. When using a moving object detection method, the detection unit 222 can be configured in the same manner as the moving object detection unit 12 of the learning data generation device 10. When using machine learning, a machine learning model such as the object detection machine learning model shown in FIG. 17 and the skeleton detection machine learning model shown in FIG. 18 can be used. Note that when a single individual cattle is included in the image, detection of the individual cattle is not necessary, and position identification, which will be described later, may be performed on the acquired image.

[0095] <Identification unit 223> The identification unit 223 identifies the individual cattle detected by the detection unit 222. The identification unit 223 is configured in the same manner as the identification unit 122 of the individual identification device 120 described above. Therefore, the description of the identification unit 223 will be cited from the description of the identification unit 122 of the individual identification device 120, and a detailed description thereof will be omitted. Note that when a single individual cattle is included in the image, it is not necessary to identify the individual cattle, and position identification, which will be described later, may be performed on the acquired image.

[0096] <Location specifying section 224> The position specifying unit 224 specifies the position of the identified body part of the individual. The positions of the body parts of the individual specified by the position specifying unit 224 are, for example, the nose tip, mouth, forelegs, hooves, etc. FIG. 22 shows an example of the position specifying unit 224 specifying the body parts of a bovine individual. The left side of FIG. 22 shows an example of specifying the nose tip, mouth, forelegs, and hooves of a sitting bovine individual, and the right side shows an example of specifying the nose tip, mouth, forelegs, and hooves of a standing bovine individual. Note that the number of body parts of the individual to be specified is not limited to this and may be fewer or more.

[0097] The position identification unit 224 may identify the position of a body part of an individual using machine learning. For example, an image including an individual is used as input data, and an image acquired by the image acquisition unit is input as input data to a position identification model that has been machine-learned to identify the position of the body part of the individual in the image, thereby identifying the output position. By inputting an image including an individual into the position identification model, an image in which the body part of the individual is identified can be output, as shown in FIG. 22.

[0098] The position identifying unit 224 may identify the position by acquiring the coordinates of the position of the body part of the individual. The position identifying unit 224 acquires the position coordinates of the body part of the individual in the image. The position identifying unit 224 may convert the acquired position coordinates into spatial coordinates of the space in which the individual cow exists. The position identifying unit 224 acquires position coordinates in multiple frames of images captured at different times.

[0099] The position identification unit 224 may identify the position by referring to the posture of the individual output by inputting the detected image data of the individual as input data into a posture estimation model that has been machine-learned to detect the skeleton of the individual from an image representing a portion of the individual's body and estimate the posture. The posture estimation model used in the position identification unit 224 is the same as the posture estimation model shown in FIG. 20. The position identification unit 224 may also use a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a portion of the individual's body that is used in the enclosing unit 13 of the learning data generation device 10. This makes it possible to identify the individual's body parts and obtain position coordinates even when there is an obstruction in the image and only a portion of the individual's body is included.

[0100] Note that when the image data includes only a portion of the individual's body, the position identification unit 224 may identify the individual's body part in the image of the portion of the individual's body and acquire the position coordinates, without generating an image representing the entire individual's body. In this case, for example, the individual's body part may be identified using a position identification model that has been trained to be able to identify the individual's body part from the image of the portion of the individual's body. Behavioral analysis can be performed even if the image of the portion of the individual's body is used as is, without restoring an image of the entire body from the image of the portion of the individual's body.

[0101] <Information acquisition unit 225> The information acquisition unit 225 acquires time-series information that represents changes in the identified positions within a predetermined time. The time-series information acquired by the information acquisition unit 225 is information that represents changes in parts of the individual cow obtained from successive images within a predetermined time, and may be the average distance between each part and the average displacement between image frames of each part, which are obtained based on the position coordinates acquired by the position identification unit 224.

[0102] <Behavioral Analysis Department 226> The behavioral pattern analysis unit 226 analyzes the behavioral pattern of the individual cattle by referring to the acquired time-series information. By referring to the time-series information, the behavioral pattern analysis unit 226 can detect the movements of the individual cattle and can also track the movements of the individual cattle. This makes it possible to analyze the behavioral pattern from the movements and movements of the individual cattle at a predetermined timing or within a predetermined time period. The behavioral pattern analysis unit 226 stores the analysis results of the behavioral patterns of the individual cattle in the analysis result storage unit 217.

[0103] The behavioral patterns of individual cattle analyzed by the behavioral pattern analysis unit 226 are types of behavior, such as lying down resting, lying down ruminating, standing up resting, standing up ruminating, feeding, etc. As shown in Fig. 23 , by previously acquiring information that associates time-series information obtained from consecutive images within a predetermined time period with the behavioral patterns of individual cattle corresponding to the time-series information, it is possible to analyze the behavioral patterns of individual cattle based on the time-series information.

[0104] The behavioral pattern analysis unit 226 may analyze the behavioral pattern of an individual cow using an analytical model constructed using training data in which time-series information is associated with the behavioral patterns of the individual cow. By inputting time-series information as input data into such an analytical model, the behavioral pattern of the individual cow is output. As shown in FIG. 23 , the behavioral pattern analysis unit 226 may display the analyzed behavioral pattern of the individual cow on a display unit (not shown) together with an image of the individual cow. Furthermore, when the image data contains only a portion of the individual's body, the behavioral pattern analysis unit 226 may analyze the individual's behavioral pattern using only the information on the position coordinates of the identified body part. In this case, the behavioral pattern of the individual may be analyzed using an analytical model trained to be able to analyze the behavioral pattern even if there is missing data in the position coordinates of the body part.

[0105] <Learning Data Generator 210> The learning data generation device 210 included in the behavioral analysis system 200 includes an image acquisition unit 11 that acquires images including individuals, a detection unit 212 that detects the individuals in the acquired images, an identification unit 213 that identifies the body region of the detected individuals, an extraction unit 14 that extracts a body region image of the identified body region, and a learning data generation unit 15 that generates learning data for a discrimination model that identifies individuals from the extracted body region image. The learning data generation device 210 further includes a learning data storage unit 16 that stores the generated learning data.

[0106] The detection unit 212 detects individuals in an image using a moving object detection method, a method using machine learning, etc. Methods for detecting individuals in an image using machine learning include a method for detecting objects using machine learning and a method for detecting the skeleton of an individual. The identification unit 213 identifies the body region of the detected individual using a method for adding a frame to a body region including the entire body of the detected individual, a method using image segmentation, etc.

[0107] The image acquisition unit 11, extraction unit 14, and training data generation unit 15 are the same as those components in the training data generation device 10 shown in Fig. 1. That is, the training data generation device 210 has the same configuration as the training data generation device 10. Therefore, the description of the training data generation device 10 will be used to explain the training data generation device 210, and a detailed description thereof will be omitted.

[0108] The training data generation device 210 can be used not only to generate training data for constructing a discrimination model, but also to generate training data for constructing an analysis model that the behavioral type analysis unit 226 uses to analyze behavioral types.

[0109] <Learning device 100> The learning device 100 constructs a discrimination model, an analytical model, and the like using the training data generated by the training data generation device 210. The learning device 100 included in the behavior-based analysis system 200 is the same as the learning device 100 shown in Fig. 10. Therefore, the description of the learning device 100 included in the behavior-based analysis system 200 will be cited, and a detailed description thereof will be omitted.

[0110] <Behavioral analysis processing> An example of the behavior-based analysis process performed by the behavior-based analysis device 220 will be described with reference to Fig. 24. Fig. 24 is a flowchart showing the behavior-based analysis process performed by the behavior-based analysis device 220 of Fig. 23.

[0111] First, in step S241, the image acquisition unit 221 acquires an image including an individual cow. Note that the behavioral analysis process by the behavioral analysis device 220 may be started by accepting input of captured image data from the camera 20, or by accepting an operation input from the user to start the behavioral analysis process.

[0112] Next, the detection unit 222 inputs the image data into a skeleton detection machine learning model and detects individual cattle in the image data (step S242).Then, the identification unit 223 inputs the image data of the detected individual cattle into an identification model, obtains individual identification information of the individual cattle, and identifies the individual cattle (step S243).

[0113] The position identification unit 224 inputs the image data of the identified individual cow into a position identification model and acquires position coordinates that identify body parts of the individual cow (step S244).The information acquisition unit 225 then acquires the identified position coordinates for several consecutive frames and calculates the average distance between each part and the average displacement between frames of each part as time-series information (step S245).

[0114] The behavioral pattern analysis unit 226 inputs the time-series information into the analysis model and analyzes the behavioral pattern of the individual cow (step S246). The behavioral pattern analysis unit 226 stores the analysis results in the analysis result storage unit 227 and ends the behavioral pattern analysis process.

[0115] In this way, the behavioral analysis device 220 can analyze the behavioral patterns of individuals by identifying the positions of body parts of the individual from image data and tracking changes in those positions. This allows for labor-saving and easy behavioral analysis, and can also detect behavioral patterns that require understanding of actions such as rumination.

[0116] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0117] 10 Learning data generation device, 11 Image acquisition unit, 12 Moving object detection unit, 13 Frame enclosing unit, 14 Extraction unit, 15 Learning data generation unit (generation unit), 100 Learning device, 101 Learning data acquisition unit, 102 Learning unit, 120 Individual identification device, 121 Image acquisition unit, 122 Identification unit, 140 Behavioral analysis device, 141 Identification image acquisition unit, 142 analysis unit, 200 behavioral analysis system, 220 behavioral analysis device, 221 image acquisition unit, 222 detection unit, 223 identification unit, 224 position identification unit, 225 information acquisition unit, 226 behavioral analysis unit

Claims

[Claim 1] A behavior analysis device for analyzing the behavior of an individual, comprising: an image acquisition unit that acquires an image including an individual; an identification unit that outputs individual identification information of an individual included in an image by inputting the acquired image data as input data into an identification model that distinguishes one individual included in a population of individuals to be identified from other individuals, the identification model being constructed by machine learning to identify individuals included in an image using learning data generated by a learning data generation device; an identified image acquisition unit that acquires time-series images of identified individuals; an analysis unit that refers to the acquired image and analyzes the behavior of the individual; It is equipped with The learning data generation device includes: an image acquisition unit that acquires an image including an individual; a detection unit that detects the individual in the acquired image; a body region specifying unit for specifying a body region including the entire body of the detected individual; an extraction unit that extracts a body region image of the identified body region; a generation unit that generates the learning data from the extracted body region image; Equipped with Behavior analysis device.

Images