Behavior pattern analysis apparatus, behavior pattern analysis method, behavior pattern analysis program, and recording medium

The behavioral analysis system addresses the high cost and labor of existing cow monitoring technologies by using machine learning to identify and analyze individual cow behavior, providing efficient and versatile monitoring across various herds.

JP2024163198AActive Publication Date: 2024-11-21NAT AGRI & FOOD RES ORG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024150845
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-07-01
Filing Date
2024-09-02
Publication Date
2024-11-21
Estimated Expiration
2041-07-01

AI Technical Summary

Technical Problem

Existing technologies for monitoring individual cow behavior in large herds are costly and labor-intensive, and video-based analysis systems are not suitable for individual cow identification and behavior analysis.

Method used

A behavioral analysis system using machine learning to identify and analyze individual cow behavior through image acquisition, detection, position identification, and behavior type analysis, utilizing a learning data generation device to construct an individual identification model and a behavior analysis device to track and analyze cow behavior.

Benefits of technology

Enables efficient, versatile, and cost-effective behavior analysis of multiple cows, capable of identifying individuals and tracking their behavior types, including rumination and feeding, regardless of herd composition and photographing conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163198000001_ABST
    Figure 2024163198000001_ABST
Patent Text Reader

Abstract

To achieve a behavior analysis system from a video which has high versatility and is applicable to various individuals.SOLUTION: A behavior pattern analysis apparatus (220) includes: an image acquisition unit (221) for acquiring an image including an individual; a detection unit (222) for detecting the individual in the image; a position specification unit (224) for specifying a position of a body portion of the detected individual; an information acquisition unit (225) for acquiring time series information showing a change in predetermined time of the specified position; and a behavior pattern analysis unit (226) for analyzing a behavior pattern of the individual with reference to the acquired time series information.SELECTED DRAWING: Figure 21
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a behavioral analysis device, a behavioral analysis method, a behavioral analysis program, and a recording medium. [Background technology]

[0002] To develop a labor-saving cow herd management technique that corresponds to the large-scale dairy farming, a method that can automatically monitor the behavior of each individual cow in the herd is required. As a technique for monitoring each individual cow in the herd, a technique that monitors each cow by attaching a wearable sensor such as an accelerometer to each cow has been put to practical use. One example of such a technique is a technique that attaches a collar-type sensor to the cow to grasp the amount of activity of the cow, calculates the time of activity, rumination, and rest, and detects cows that require attention such as estrus or suspected disease from the information.

[0003] However, the larger the cattle herd, the higher the introduction costs of this technology and the greater the labor required to manage the equipment. The cost of introducing and operating this technology is, for example, several tens of thousands of yen per collar-type sensor to monitor the activity of cattle, and several thousand yen per cattle per year in usage fees.

[0004] In contrast, technology for monitoring cows based on video analysis has the advantage of reducing costs and being easy to manage, since it is not necessary to attach sensors to the cows. Examples of technology for monitoring cows based on video analysis include the technologies described in Non-Patent Documents 1 and 2. Non-Patent Document 1 shows that it is possible to detect hoof disease by analyzing the cow's gait from images of the cow using a depth camera. In addition, Non-Patent Document 2 describes the detection of changes in posture, etc., which are a sign of parturition, from images of the cow.

[0005] Furthermore, examples of technologies for identifying and tracking individuals include those described in Patent Documents 1 and 2. Patent Document 1 describes a configuration for detecting people moving within a specific area, acquiring still images of the people, determining whether they are residents or non-residents, and tracking their behavior over time. Patent Document 2 describes a configuration for detecting moving objects in a video and extracting features for identifying people. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2017-224249 [Patent Document 2] Patent Publication No. 2019-169843 [Non-patent literature]

[0007] [Non-Patent Document 1] Sho Sunagawa et al., Information Processing Society of Japan Technical Report, Vol. 2017-CVIM-206, No. 2 (2017) [Non-Patent Document 2] Yusuke Okimoto et al., Proceedings of the 32nd National Conference of the Japanese Society for Artificial Intelligence (2018) Summary of the Invention [Problem to be solved by the invention]

[0008] However, the techniques described in Non-Patent Documents 1 and 2 are intended for monitoring a single individual, and are therefore not suitable for analyzing the behavior of multiple individual cows.

[0009] On the other hand, the technologies described in Patent Documents 1 and 2 monitor multiple individuals, and although they describe the use of a machine-learned model for individual identification, they make no mention of the collection of learning data for machine learning. In order to make the technology an individual identification technology that can be used for a variety of cattle groups with different individual configurations and photography conditions, it is necessary to have an individual identification system that uses a trained model that can be easily constructed for each cattle group.

[0010] One aspect of the present invention has been made to solve the above-mentioned problems, and its purpose is to realize a highly versatile system for analyzing behavior from video that can be applied to a variety of individuals. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention includes the following aspects. 1) The behavioral analysis device includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the image, a position identification unit that identifies a position of a body part of the detected individual, an information acquisition unit that acquires time series information indicating a change in the identified position within a predetermined period of time, and a behavioral analysis unit that analyzes the behavioral pattern of the individual by referring to the acquired time series information. 2) In the behavioral analysis device described in 1), the behavioral type of the individual analyzed by the behavioral analysis unit is selected from the group consisting of recumbent resting, recumbent rumination, standing resting, standing rumination, and feeding. 3) In the behavioral analysis device described in 1) or 2), the position identification unit identifies the position output by inputting the image acquired by the image acquisition unit as input data into a position identification model that has been machine-trained to identify the position of a body part of the individual in an image including the individual. 4) In the behavioral analysis device described in any one of 1) to 3), the position identification unit acquires coordinates of a position of a body part of the individual. 5) In the behavioral analysis device described in any one of 1) to 4), the behavioral analysis unit inputs time-series information acquired by the information acquisition unit as input data into an analysis model that has been machine-learned to estimate a behavioral type corresponding to the time-series information, and acquires, as an analysis result, a behavioral type to be output. 6) In the behavioral analysis device described in any one of 1) to 5), the position identification unit identifies the position by referring to the posture of the individual that is output by inputting image data of the individual as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture. 7) The behavioral analysis device according to any one of 1) to 6) further includes an identification unit that identifies the individual detected by the detection unit. 8) The behavioral analysis method includes an image acquisition step of acquiring an image including an individual, a detection step of detecting the individual in the image, a position identification step of identifying a position of a body part of the detected individual, an information acquisition step of acquiring time series information showing a change in the identified position within a predetermined period of time, and a behavioral analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time series information. 9) The behavioral analysis program is a behavioral analysis program for causing a computer to function as the behavioral analysis device described in any one of 1) to 7), and causes the computer to function as the image acquisition unit, the detection unit, the position identification unit, the information acquisition unit, and the behavioral analysis unit. 10) A computer-readable recording medium having recorded thereon the behavioral analysis program described in 9). Effect of the Invention

[0012] According to one aspect of the present invention, a highly versatile video behavior analysis system that can be applied to a variety of individuals can be realized. [Brief description of the drawings]

[0013] [Figure 1]1 is a block diagram showing a configuration of a main part of a learning data generating device according to an embodiment of the present invention; [Diagram 2] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Diagram 3] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 4] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Diagram 5] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 6] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 7] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 8] 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. [Figure 9] 4 is a flowchart showing a learning data generation process performed by the learning data generation device. [Figure 10] 1 is a block diagram showing a configuration of a main part of a learning device according to an embodiment of the present invention; [Figure 11] 4 is a flowchart showing a learning process performed by the learning device. [Figure 12] 1 is a block diagram showing a configuration of a main part of an individual identification device according to an embodiment of the present invention; [Figure 13] 4 is a flowchart showing an individual identification process performed by the individual identification device. [Figure 14] 1 is a block diagram showing a configuration of a main part of a behavior analysis device according to an embodiment of the present invention. [Figure 15] 13 is a flowchart showing a behavior analysis process performed by the behavior analysis device. [Figure 16] FIG. 1 is a diagram illustrating an example of individual identification used in a learning data generation device according to an embodiment of the present invention. [Figure 17]FIG. 1 is a diagram illustrating an example of an object detection machine learning model used to detect individuals in an image in a learning data generation device according to an embodiment of the present invention. [Figure 18] FIG. 13 is a diagram illustrating an example of a skeleton detection machine learning model used to detect an individual in an image in a learning data generation device according to one embodiment of the present invention. [Figure 19] 1A to 1C are diagrams illustrating an example of a form in which a body region is identified in a learning data generation device according to an aspect of the present invention. [Figure 20] 1 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from an image showing a part of the body in a learning data generation device according to an embodiment of the present invention. FIG. [Figure 21] 1 is a block diagram showing a configuration of a main part of a behavioral analysis system according to an embodiment of the present invention. [Figure 22] FIG. 2 is a diagram illustrating examples of body parts of an individual identified by the behavioral analysis device. [Diagram 23] FIG. 2 is a diagram illustrating an example of an individual's behavior type analyzed by the behavior type analysis device. [Figure 24] 13 is a flowchart showing a behavioral analysis process performed by the behavioral analysis device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Learning Data Generator] A learning data generation device according to one embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, a moving object detection unit that detects a moving object in the acquired image, a framing unit that adds a frame to a body region of the detected moving object that includes the entire body of the individual, an extraction unit that extracts a body region image of the body region to which the frame has been added, and a generation unit that generates learning data for an individual identification model that identifies individuals from the extracted body region image.

[0015] 1 is a block diagram showing the main components of a training data generation device according to an embodiment of the present invention. The training data generation device 10 includes an image acquisition unit 11, a moving object detection unit 12, a framing unit 13, an extraction unit 14, a training data generation unit (generation unit) 15, and a training data storage unit 16. The training data generation device 10 generates training data used to build an individual identification model by machine learning.

[0016] (individual) The individual identification model that machine-learns the learning data generated by the learning data generation device 10 is a machine-learned model that identifies individuals in an image in which a plurality of individuals are photographed. The individuals identified by the individual identification model may be humans, animals, other moving objects, etc., but are preferably individuals managed in groups in a closed space. The individuals identified by the individual identification model are more preferably livestock individuals that are raised in large numbers, even more preferably cattle individuals, even more preferably dairy cow individuals, and most preferably Holstein dairy cow individuals. In the following, Holstein dairy cow individuals will be described as an example of individuals to be identified, but the present invention is not limited thereto, and other dairy cow individuals or other animal individuals can be applied in the same manner as Holstein dairy cow individuals by using differences in patterns or body shapes, or simple markers, etc. For example, the present invention can also be applied to Nishikigoi, pigs, cats, dogs, etc. As an example of a configuration that uses a simple marker, a configuration in which a mark such as a numerical value is attached to a collar can be given as shown in FIG. 16. FIG. 16 is a diagram for explaining an example of individual identification used in a learning data generation device according to one aspect of the present invention. By using such simple markers, it is possible to easily identify individuals even of species that are difficult to identify by pattern, and the present invention can be applied to individuals of a variety of species.

[0017] The multiple animals may be free-roaming indoor animals as shown in Fig. 2, or may be tethered animals such as chained animals. Fig. 2 is a diagram for explaining the learning data generation process of the learning data generation device according to one aspect of the present invention. The multiple animals may be animals kept in a controlled area outdoors. In this embodiment, the case where the animals are cows will be described as an example.

[0018] (Image acquisition unit 11) The image acquisition unit 11 acquires images including individual cattle. The image acquisition unit 11 acquires images including the individual cattle from the camera 20 that captured the image of the individual cattle via an input / output unit (not shown). The image acquisition unit 11 acquires images including the individual cattle from the camera 20 by wireless communication or wired communication. The image acquisition unit 11 may acquire an image every time the individual cattle is captured by the camera 20, or may acquire an image at predetermined time intervals.

[0019] The camera 20 may photograph the individual cattle from above or from diagonally above. In the learning data generation device 10, not only images of the individual cattle photographed from above but also images photographed from diagonally above can be suitably used. As an example, when the individual cattle are kept indoors, the camera 20 may be installed on the ceiling or on a wall. Also, when the individual cattle are kept outdoors, the camera 20 may be fixed to a pole, a fence, or the like.

[0020] The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken by a thermo camera or the like, or a three-dimensional image obtained by a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken by a laser sensor, it is possible to obtain an image at night without using lighting. The image including the individual cattle may be a still image or a video. The image acquisition unit 11 preferably acquires images including the individual cattle from a plurality of cameras 20, and more preferably acquires images of the individual cattle from different directions by the plurality of cameras 20. The image acquisition unit 11 sends the acquired image including the individual cattle to the moving object detection unit 12.

[0021] (Motion detection unit 12) The moving object detection unit 12 detects a moving object in the acquired image including the individual cow. The moving object detection unit 12 detects the movement of the object in the image including the individual cow and detects the moving object as a moving object. The moving object detection unit 12 detects a moving object in the image, for example, by inter-frame difference processing in which a comparison is made between a plurality of image frames captured at different times, background difference processing in which a captured image is compared with a background image acquired in advance, or the like. The moving object detection unit 12 acquires position information of the detected moving object in the image and sends it to the enclosing unit 13.

[0022] (Boxed section 13) The framing unit 13 frames a body region including the entire body of a bovine individual among the detected moving objects. When the detected moving object is a bovine individual, the framing unit 13 frames the body region of the bovine individual. The framing unit 13 identifies the body region of the bovine individual in an image of a partial region corresponding to the acquired position information of the moving object, and frames the body region.

[0023] If a frame is added to an image of a partial region corresponding to the position information of a moving object without specifying the body region of the individual cow, the frame may be added only to a part of the body, such as the head or tail of the individual cow, as shown in Figure 3. Also, the frame may be added to a moving object other than the individual cow.

[0024] On the other hand, the framing unit 13 identifies and frames the body region of the individual cow in the image of the partial region corresponding to the position information of the moving object, so that the entire body of the individual cow is framed inside the frame, as shown in Fig. 4. Fig. 4 is a diagram for explaining the learning data generation process of the learning data generation device according to one embodiment of the present invention.

[0025] The body region of the individual cow in the enclosing unit 13 can be specified by, for example, a method using a trained model that has been machine-learned to specify the body region of the individual cow, or a method such as feature point matching or pattern matching. That is, the enclosing unit 13 gives a frame to the body region output by inputting image data of the detected moving object as input data to a discrimination model that has been machine-learned to extract a body region including the entire body of the individual cow from an image including the individual cow. Such a machine-learned discrimination model can be obtained by applying a known learning algorithm such as a support vector machine, a random forest, a k-means method, or a convolutional neural network to learning data as shown in FIG. 5, which is accumulated with images of the entire body of thousands to tens of thousands of individual cows. FIG. 5 is a diagram for explaining a learning data generation process of the learning data generation device according to one embodiment of the present invention.

[0026] Furthermore, in cases where only a part of the individual cow is included in the captured image, where the individual cows are overlapping, or where a part of the individual cow is hidden by an obstacle such as a pillar or a wall, the framing unit 13 generates a body region image from an image representing a part of the individual cow and adds a frame to the generated body region image. That is, the framing unit 13 sets the image representing the entire body of the individual, which is output by inputting image data of the detected moving object as input data to a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a part of the individual's body, as the body region, and adds a frame to the body region.

[0027] As such a generative model, a known generative model such as a variational autoencoder (VAE) or a generative adversarial network (GAN) can be used. For example, when a variational autoencoder is used, as shown in FIG. 6, a numerical value representing the feature of image data is obtained from image data of a part of a body of a cow individual by convolution of an encoder. Then, an image of the entire body of the cow individual is restored by deconvoluting the numerical value representing the feature. FIG. 6 is a diagram for explaining a learning data generation process of a learning data generation device according to an embodiment of the present invention. For example, as shown in FIG. 6, a known generative model is applied to learning data that accumulates images of the entire body of thousands to tens of thousands of cow individuals, thereby obtaining a machine-learned generative model. In this way, by using a generative model that has been trained to restore a body region image of the entire body from an image of a part of the body of a cow individual, an image of the entire body of the cow individual can be restored using an image of a part of the body of the cow individual as input image data. Note that the data output from the generative model in the example shown in FIG. 6 may be only a frame surrounding the entire body, and patterns, colors, etc. may not be output.

[0028] 7 and 8 are diagrams for explaining the learning data generation process of the learning data generation device according to one embodiment of the present invention. When image data representing a part of the body of a cow individual is input to the machine-learned generation model as shown in FIG. 7, a numerical value Z representing the characteristics of the image data is obtained by convolution of the decoder. The numerical value Z is a numerical value representing the characteristics of the input image, for example, a numerical value representing information such as the scale of the image and the range of the entire body that the area included in the image corresponds to. Then, by restoring the obtained numerical value Z by deconvolution, a body region image representing the body region of the cow individual is generated and output. Also, when image data representing a part of the body of a cow individual including a feature not included in the learning data is input to this machine-learned generation model as shown in FIG. 7, a body region image of the cow individual is generated and output. In this way, even when unknown cow image data is input to the generation model, data surrounding the entire body of the cow individual is output. In this way, even when image data representing a part of the body of a cow individual including a feature not included in the learning data is input to the generation model, data surrounding the entire body of the cow individual is output. Furthermore, even when the scale of an image showing a part of a body of an individual cow is larger than that of FIG. 7, as in the input data shown in FIG. 8, an image of the body region of the individual cow surrounded by a frame with a size according to the scale of the input image is output.

[0029] In this way, when the image of the moving object is only part of the cow, the framing unit 13 generates the body area of ​​the cow and applies a frame by using a generative model trained by machine learning to generate an image representing the entire body of the cow from an image representing part of the body of the cow.

[0030] (Extraction part 14) The extraction unit 14 extracts the body region image with the frame added. The extraction unit 14 removes the image outside the frame from the image captured by the camera 20, and cuts out the body region image inside the frame. The extraction unit 14 sends the extracted body region image to the learning data generation unit 15.

[0031] (Learning data generation unit 15) The learning data generating unit 15 generates learning data for an individual identification model that identifies an individual from the extracted body area images. The learning data may be unlabeled unsupervised data, and may include, for example, hundreds to tens of thousands of body area image data. The learning data generating unit 15 may generate learning data by classifying and storing the body area images for each individual. The learning data generating unit 15 may generate learning data from the stored body area images when a predetermined number of body area images have been stored or when an instruction to generate learning data is received from a user.

[0032] The classification of the body area images for each individual by the learning data generating unit 15 can be performed by, for example, referring to information such as the similarity of the body area images, the positional relationship with previous and next images in a time series, etc. Such classification of the body area images for each individual may be performed using a classification estimation model that is machine-learned to estimate the body area image based on the similarity of the body area image from the body area image or the positional relationship with previous and next images in a time series.

[0033] The learning data generating unit 15 generates learning data corresponding to the population identified by the individual identification model. For example, when the individual identification model that performs machine learning using the learning data identifies population X including 10 bovine individuals from bovine individual No. 1 to bovine individual No. 10, the learning data generating unit 15 generates learning data including body region image data of the 10 bovine individuals included in population X. The learning data generating unit 15 stores the generated learning data in the learning data storage unit 16 in association with information on the corresponding population.

[0034] (Learning data storage unit 16) The generated learning data is stored in the learning data storage unit 16. The learning data storage unit 16 may also store data such as images acquired by the image acquisition unit 11 and position information in the images of moving objects detected by the moving object detection unit 12. The learning data storage unit 16 may also store a program indicating a series of processes required for the operation performed by the learning data generation device 10.

[0035] (Learning data generation process) An example of the learning data generation process performed by the learning data generation device 10 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the learning data generation process performed by the learning data generation device 10 of Fig. 1.

[0036] First, in step S91, the image acquisition unit 11 acquires an image including an individual cow. The learning data generation process by the learning data generation device 10 may be started by receiving an input of captured image data from the camera 20.

[0037] Next, the moving object detection unit 12 detects a moving object in the acquired image including the individual cow (step S92). Then, in step S93, the framing unit 13 frames the body region of the individual cow among the detected moving objects. Next, in step S94, the extraction unit 14 extracts the framed body region image. Then, the learning data generation unit 15 generates learning data from the body region image (step S94). The learning data generation unit 15 stores the generated learning data in the learning data storage unit 16, and ends the learning data generation process.

[0038] In this way, the learning data generation device 10 automatically generates learning data used to construct an individual identification model by machine learning by extracting and storing body region images of individual cattle detected as moving objects. This makes it easy to generate learning data for constructing an individual identification model. Therefore, learning data can be easily generated for each individual group for various individual groups with different individual configurations and shooting conditions. In addition, the learning data generated by the learning data generation device 10 includes body region images of individual cattle in various orientations. Therefore, by performing individual identification using an individual identification model trained using such learning data, individual identification becomes possible regardless of the orientation of the individual cattle in the image to be identified.

[0039] In the machine learning of image recognition, a problem has been that it takes time and effort to prepare learning data. The learning data generating device 10 can automatically accumulate learning data used in the machine learning of the individual identification model. This makes it possible to easily build an individual identification model that corresponds to each dairy farm site, which has different individual compositions of cattle populations and different shooting conditions, and realizes general-purpose use of the individual identification model.

[0040] [Learning device] A learning device according to one embodiment of the present invention includes a learning data acquisition unit that acquires learning data generated by a learning data generation device according to one embodiment of the present invention, and a learning unit that constructs an individual identification model that identifies individuals by machine learning using the acquired learning data to identify individuals contained in an image.

[0041] 10 is a block diagram showing the configuration of the main parts of a learning device according to an embodiment of the present invention. The learning device 100 includes a learning data acquisition unit 101, a learning unit 102, and a learning model storage unit 103. The learning device 100 constructs an individual identification model that identifies individuals.

[0042] (Learning data acquisition unit 101) The learning data acquisition unit 101 acquires the learning data stored in the learning data storage unit 16 of the learning data generation device 10 via an input / output unit (not shown). The learning data acquisition unit 101 acquires learning data corresponding to a population identified by an individual identification model machine-learned in the learning unit 102. For example, when the individual identification model is a trained model for identifying cattle individuals of a population X including 10 cattle individuals from cattle individual No. 1 to cattle individual No. 10, the learning data acquisition unit 101 acquires learning data corresponding to the population X. The learning data acquisition unit 101 sends the acquired learning data to the learning unit 102.

[0043] (Learning section 102) The learning unit 102 uses the acquired learning data to construct an individual identification model that identifies individuals. The learning unit 102 uses the acquired learning data to machine-learn the individual identification model by applying a known learning algorithm such as a support vector machine, a random forest, a k-means method, or a convolutional neural network.

[0044] The machine-learned individual identification model constructed in the learning unit 102 is a trained model that outputs identification information for identifying an individual by inputting image data including an individual as input data. For example, when the individual identification model identifies a cow individual in the population X, the individual identification model outputs which cow individual in the population X (cow individual No. 1, cow individual No. 2, ..., or cow individual No. 10) the cow individual in the input image data is. The learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103.

[0045] (Learning model storage unit 103) The learning model storage unit 103 stores an individual identification model that has been machine-learned. The learning model storage unit 103 may also store learning data acquired by the learning data acquisition unit 101. The learning model storage unit 103 may also store a program indicating a series of processes required for the operation of the learning device 100.

[0046] (Learning process) An example of the learning process by the learning device 100 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the learning process by the learning device 100 of Fig. 10.

[0047] First, in step S111, the learning data acquisition unit 101 acquires learning data from the learning data generation device 10. Next, the learning unit 102 uses the acquired learning data to machine-learn an individual identification model that identifies individuals (step S112). Then, the learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103 (step S113), and ends the learning process.

[0048] In this way, the learning device 100 constructs an individual identification model by machine learning using learning data generated for each population, so it is possible to easily construct an individual identification model corresponding to each dairy farm site where the individual composition of the cattle population and the shooting conditions are different, for example. As a result, the individual identification model can be used for general purposes.

[0049] [Individual Identification Device] An individual identification device according to one embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, and an identification unit that outputs individual identification information of an individual included in the image by inputting data of the acquired image as input data into the individual identification model constructed by a learning device according to one embodiment of the present invention.

[0050] 12 is a block diagram showing the main components of an individual identification device according to an embodiment of the present invention. An individual identification device 120 includes an image acquisition unit 121, an identification unit 122, and an identification image storage unit 123. The individual identification device identifies individuals in a population using an individual identification model.

[0051] (individual) The individual identified by the individual identification device 120 may be a human, an animal, another moving object, etc., but is preferably an individual managed in groups in a closed space. The individual identified by the individual identification model is more preferably a livestock individual raised in a number of animals, even more preferably a cow individual, even more preferably a dairy cow individual, and most preferably a Holstein dairy cow individual. The raised individual may be an individual that is allowed to roam freely indoors as shown in FIG. 2, or an individual that is tied to a chain or the like and kept in a tethered state. The raised individual may also be an individual that is raised in a managed area outdoors. In this embodiment, a case in which the individual is a cow individual will be described as an example.

[0052] (Image acquisition unit 121) The image acquisition unit 121 acquires an image including an individual cow. The image acquisition unit 121 acquires an image including the individual cow from the camera 20 that captured the image of the individual cow via an input / output unit (not shown). The image acquisition unit 121 acquires an image including the individual cow from the camera 20 by wireless communication or wired communication. The image acquisition unit 121 may acquire an image every time the individual cow is captured by the camera 20, or may acquire an image at predetermined time intervals.

[0053] The camera 20 may photograph the individual cow from above or from diagonally above. In the individual identification device 120, not only an image of the individual cow photographed from above but also an image photographed from diagonally above can be suitably used. As an example, when the individual cow is kept indoors, the camera 20 may be installed on the ceiling or on a wall. Also, when the individual cow is kept outdoors, the camera 20 may be fixed to a pole, a fence, or the like.

[0054] The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken by a thermo camera or the like, or a three-dimensional image obtained by a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken by a laser sensor, it is possible to obtain an image at night without using lighting. Furthermore, the image including the individual cattle may be a still image or a video. The image obtaining unit 121 sends the obtained image including the individual cattle to the identification unit 122.

[0055] (Identification unit 122) The identification unit 122 inputs the data of the acquired image as input data into the individual identification model constructed by the learning device 100, and outputs individual identification information of the individual cattle in the image. The identification unit 122 acquires the individual identification model stored in the learning model storage unit 103 of the learning device 100 via an input / output unit (not shown). The individual identification model is a machine-learned model corresponding to the individual group identified by the individual identification device 120. For example, when the individual group identified by the individual identification device 120 is population X including 10 individual cattle, from cow individual No. 1 to cow individual No. 10, the individual identification model is a machine-learned model corresponding to the population X.

[0056] The identification unit 122 associates individual identification information, which indicates which individual bovine animal the individual bovine animal contained in the acquired image is, with the image data, and stores the associated information in the identified image storage unit 123. Furthermore, the identification unit 122 may add the individual identification information to the image data and display it on a display unit (not shown), as shown in Fig. 4. For example, the identification unit 122 displays the individual bovine animal in the image by adding a bovine animal number that identifies the individual bovine animal, as shown in Fig. 4.

[0057] The image identified by the identification unit 122 may be an image obtained by extracting a body region image including the entire body of the individual cow from the image acquisition unit 121. That is, the individual identification device 120 may further include a framing unit that adds a frame to the body region of the individual cow in the image acquired by the image acquisition unit 121, and an extraction unit that extracts the body region image to which the frame has been added. The individual identification device 120 may further include a moving object detection unit that detects a moving object in the image acquired by the image acquisition unit 121. Then, the identification unit 122 outputs individual identification information of the individual cow in the image by inputting the body region image extracted by the extraction unit as input data to an individual identification model. Note that the image identified by the identification unit 122 may be the image itself acquired by the image acquisition unit 121 from the camera 20, for example, when the individual cow in the image is alone.

[0058] In this way, the individual identification device 120 may use, as the image to be subjected to individual identification, a body region image that has been subjected to the same processing as the learning data generation processing by the learning data generation device 10, instead of the image itself acquired by the camera 20. This allows the image to be subjected to individual identification to not include parts unnecessary for individual identification, enabling more accurate individual identification.

[0059] (Identification image storage unit 123) Images that have been individually identified are stored in the identification image storage unit 123. Furthermore, the identification image storage unit 123 may store a program indicating a series of processes required for the operation of the individual identification device 120.

[0060] (Individual Identification Processing) An example of the individual identification process by the individual identification device 120 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the individual identification process by the individual identification device 120 of Fig. 12.

[0061] First, in step S131, the image acquisition unit 121 acquires an image including an individual cow. Note that the individual identification process by the individual identification device 120 may be started by accepting an input of captured image data from the camera 20.

[0062] Next, the identification unit 122 reads out the trained individual identification model from the learning device 100 (step S132). Then, in step S133, the identification unit 122 inputs image data into the read individual identification model. Then, in step S134, the identification unit 122 associates the individual identification information of the individual cow output by the individual identification model with the image, stores them in the identification image storage unit 123, and ends the individual identification process.

[0063] In this way, the individual identification device 120 identifies individuals using an individual identification model corresponding to each population, so that, for example, individual identification corresponding to each dairy farm site where the individual composition of the cattle population and the shooting conditions are different is possible. Furthermore, by using the individual identification model trained by the learning device 100, individual identification is possible regardless of the orientation of the individual cattle in the image to be identified. The individual identification device 120 allows general-purpose use of the individual identification model.

[0064] [Behavior analysis device] A behavioral analysis device according to one embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, an identification unit that outputs individual identification information of an individual included in an image by inputting data of the acquired image as input data into an identification model for identifying an individual constructed by machine learning to identify an individual included in the image using learning data generated by the learning data generation device, an identification image acquisition unit that acquires time-series images of the identified individual, and an analysis unit that analyzes the behavior of the individual by referring to the acquired images. That is, the behavioral analysis device includes an identification image acquisition unit that acquires time-series images of an individual identified by the individual identification device according to one embodiment of the present invention, and an analysis unit that analyzes the behavior of the individual by referring to the acquired images.

[0065] 14 is a block diagram showing the configuration of the main parts of a behavior analysis device according to one aspect of the present invention. Behavior analysis device 140 includes identification image acquisition section 141, analysis section 142, and analysis result storage section 143. The behavior analysis device tracks and analyzes the behavior of identified individuals.

[0066] (Identification image acquisition unit 141) The identification image acquisition unit 141 acquires images of individual cattle stored in the identification image storage unit 123 of the individual identification device 120 via an input / output unit (not shown). The identification image acquisition unit 141 acquires time-series images of individual cattle identified by the individual identification device 120. The identification image acquisition unit 141 sends the acquired time-series images of individual cattle to the analysis unit 142.

[0067] (Analysis Department 142) The analysis unit 142 analyzes the behavior of the individual cow by referring to the time-series images of the individual cow. The behavior of the individual cow can be tracked by the time-series images of the individual cow. The analysis unit 142 analyzes the behavior of the individual cow within a predetermined time by tracking the behavior of the individual cow based on the time-series images of the individual cow. The analysis unit 142 analyzes, for example, the amount of movement, range of movement, and pattern of movement of the individual cow within a predetermined time by tracking the behavior of the individual cow based on the time-series images of the individual cow. The analysis unit 142 stores the analysis results of the individual cow in the analysis result storage unit 143.

[0068] The analysis unit 142 may predict the physical condition, estrus state, disease, etc. of the individual cow by analyzing the behavioral patterns of the individual cow, such as the time the individual cow sits, stands, moves around, etc. Furthermore, the analysis unit 142 may manage the feeding of the individual cow by analyzing the behavioral patterns of the individual cow, such as the number of times and time the individual cow drinks water, and the number of times and time the individual cow eats feed, etc.

[0069] The analysis unit 142 may acquire position coordinates in the image of the individual cow from the image of the individual cow, and convert the position coordinates into spatial coordinates of the space in which the individual cow exists, thereby tracking and analyzing the movement range, movement pattern, etc. of the individual cow in the space. The conversion from position coordinates to spatial coordinates can be performed, for example, by converting the acquired position coordinates into spatial coordinates of the space in which the individual cow exists, using information on a reference point whose spatial coordinates on the image are known. If the image of the individual cow referred to by the analysis unit 142 is a body region image including the entire body of the individual cow, the position coordinates of the feet of the individual cow can be acquired, and therefore the behavior of the individual cow in the space can be accurately tracked. The position coordinates or spatial coordinates of the feet of the individual cow may be displayed on a display unit (not shown) together with the body region image of the individual cow as shown in FIG. 4.

[0070] (Behavioral Analysis Processing) An example of behavior analysis processing by behavior analysis apparatus 140 will be described with reference to Fig. 15. Fig. 15 is a flow chart showing behavior analysis processing by behavior analysis apparatus 140 of Fig. 14.

[0071] First, in step S151, the identification image acquisition unit 141 acquires time-series image data of the individual cow. Next, the analysis unit 142 tracks the behavior of the individual cow using the time-series images of the individual cow and analyzes the behavior of the individual cow (step S152). The analysis unit 142 stores the analysis result in the analysis result storage unit 143 and ends the behavior analysis process.

[0072] In this way, the behavior analysis device 140 analyzes the behavior of individuals by tracking images of individuals identified using an individual identification model corresponding to each population, so that accurate and easy individual behavior analysis is possible for each dairy farm site where the individual composition of the cattle population and the shooting conditions are different, for example. Note that a behavior analysis system including the learning data generation device 10 shown in Fig. 1, the learning device 100 shown in Fig. 10, the individual identification device 120 shown in Fig. 12, and the behavior analysis device 140 shown in Fig. 14 are also included in the scope of the present invention.

[0073] [Method for generating learning data and method for analyzing behavior] A training data generation method according to an aspect of the present invention includes an image acquisition step of acquiring an image including an individual, a moving object detection step of detecting a moving object in the acquired image, a framing step of framing a body region including the entire body of the individual among the detected moving objects, an extraction step of extracting a body region image of the body region to which the framing has been applied, and a generation step of generating training data for a discrimination model that identifies the individual from the extracted body region image. As an example, the training data generation method according to an aspect of the present invention is realized by the training data generation device according to an aspect of the present invention described above. Therefore, the description of the training data generation method according to an aspect of the present invention is similar to the description of the training data generation device according to an aspect of the present invention described above.

[0074] A behavioral analysis method according to an aspect of the present invention includes an image acquisition step of acquiring an image including an individual, an identification step of inputting data of the acquired image as input data into an identification model for identifying an individual, which is constructed by machine learning to identify an individual included in the image using learning data generated by the learning data generation method, thereby outputting individual identification information of the individual included in the image, an identification image acquisition step of acquiring time-series images of the identified individual, and an analysis step of analyzing the behavior of the individual by referring to the acquired images. As an example, the behavioral analysis method according to an aspect of the present invention is realized by the behavioral analysis device according to the aspect of the present invention described above. Therefore, the description of the behavioral analysis method according to an aspect of the present invention is similar to the description of the behavioral analysis device according to the aspect of the present invention described above.

[0075] [Software implementation example] The control blocks of the learning data generation device 10, the learning device 100, the individual identification device 120, and the behavior analysis device 140 may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0076] In the latter case, each of the above-mentioned devices includes a computer that executes instructions of a program, which is software that realizes each function. The computer includes, for example, one or more processors, and a computer-readable recording medium that stores the program. The object of the present invention is achieved by the processor reading the program from the recording medium and executing it in the computer. The processor can be, for example, a CPU (Central Processing Unit). The recording medium can be a "non-transient tangible medium", such as a ROM (Read Only Memory), as well as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc. The device may further include a RAM (Random Access Memory) that expands the program. The program may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0077] In other words, the learning data generation device, learning device, individual identification device, and behavioral analysis device 140 according to each aspect of the present invention may be realized by a computer. In this case, the program for realizing each of the devices on a computer by causing the computer to operate as each part (software element) of each of the devices, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention.

[0078] Specifically, the following programs also fall within the scope of the present invention: A program for causing a computer to function as a learning data generation device according to each aspect of the present invention, the program executing an image acquisition step of acquiring an image including an individual, a moving object detection step of detecting a moving object in the acquired image, a framing step of adding a frame to a body region including the entire body of the individual among the detected moving objects, an extraction step of extracting a body region image of the body region to which the frame has been added, and a generation step of generating learning data for a discriminative model that identifies individuals from the extracted body region image: A program for causing a computer to function as a behavioral analysis device according to each aspect of the present invention, the program comprising: an image acquisition step of acquiring an image including an individual; an identification step of inputting data of the acquired image as input data into an identification model for identifying an individual, the identification model being constructed by machine learning to identify an individual included in the image using learning data generated by the learning data generation device according to each aspect of the present invention, thereby outputting individual identification information of the individual included in the image; an identification image acquisition step of acquiring images in time series of the identified individual; and an analysis step of analyzing behavior of the individual by referring to the acquired images, the program comprising: A program for causing a computer to function as a learning device according to each aspect of the present invention, the program comprising: a learning data acquisition step for acquiring learning data generated by a learning data generation device according to each aspect of the present invention; and a learning step for constructing an identification model for identifying individuals by machine learning using the acquired learning data to identify individuals included in an image. The program comprises: A program for causing a computer to function as an individual identification device relating to each aspect of the present invention, the program comprising: an image acquisition step for acquiring an image including an individual; and an identification step for inputting data of the acquired image as input data into the identification model constructed by the learning device relating to each aspect of the present invention, thereby outputting individual identification information of the individual included in the image.

[0079] [Modifications] A learning data generation device according to another aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, a body region identification unit that identifies a body region including the entire body of the detected individual, an extraction unit that extracts a body region image of the identified body region, and a generation unit that generates learning data for a discrimination model that identifies individuals from the extracted body region image.

[0080] That is, in the learning data generating device, the method of detecting an individual in an image is not limited to a moving object detection method, and may be realized by other methods of detecting an individual. Another example of detecting an individual in an image is a method using machine learning. Therefore, in the learning data generating device, a configuration in which a detection unit that detects an individual in an image using machine learning instead of a moving object detection unit is also included in the scope of the present invention.

[0081] When detecting an individual in an image using machine learning, it is preferable to create a machine learning model suitable for detecting the individual depending on the shooting direction of the image. For example, when using an image in which an individual is shot from diagonally above, a machine learning model generated to be suitable for detecting the individual in the image shot from diagonally above is used. Methods for detecting an individual in an image using machine learning include a method for detecting an object using machine learning and a method for detecting the skeleton of an individual. Such a detection unit can also be used in an individual identification device according to one aspect of the present invention.

[0082] When using a machine learning model created to detect an object from an image, image data of an individual cow is input as input data to the object detection machine learning model, and a detection result of the individual cow part in the image is output, as shown in Figure 17. When using a machine learning model created to detect a skeleton from an image, image data of an individual cow is input as input data to the skeleton detection machine learning model, as shown in Figure 18. As a result, the skeleton positions of individual cows in the image are detected and organized for each individual cow, and a detection result of the individual cow part in the image is output.

[0083] In addition, in the learning data generation device, the method of identifying the body region including the entire body of the detected individual is not limited to the method of adding a frame, and may be realized by other methods of identifying the body region of the individual. As another example of identifying the body region of the individual, a method using image segmentation can be mentioned as shown in FIG. 19. FIG. 19 is a diagram for explaining an example of a form of identifying the body region in the learning data generation device according to one embodiment of the present invention. As shown in FIG. 19, the body region is identified by image segmentation in an image including a cow individual, and, for example, position coordinates indicating only the identified body region range are obtained. In this way, a configuration in which a body region identification unit that identifies the body region of the individual by image segmentation instead of a frame unit is used in the learning data generation device is also included in the scope of the present invention. In addition, such a body region identification unit can also be used in the individual identification device according to one embodiment of the present invention.

[0084] Furthermore, in the learning data generation device, the method of identifying an image of the entire body from an image representing a part of the individual's body is not limited to the method of restoring images representing other parts of the body, and may be realized by other methods of identifying an image of the entire body from an image representing a part of the body. Another example of identifying an image of the entire body from an image representing a part of the body is a method of referring to a posture estimated based on skeletal detection using machine learning.

[0085] An example of identifying an image of the entire body from an image showing a part of the body by referring to a posture estimated based on skeleton detection is shown in Fig. 20. Fig. 20 is a diagram illustrating an example of a posture estimation model used for identifying an image of the entire body from an image showing a part of the body in a learning data generation device according to one embodiment of the present invention. As shown in Fig. 20, when image data including only a part of an individual cow is acquired, the position of the part not included in the image (invisible part) is identified by referring to a posture estimated from the part included in the image (visible part).

[0086] For example, in the image data of FIG. 20, only seven parts are visible: the nose, the left eye, the right eye, the left ear, the right ear, the left shoulder, and the left hip. When this image data is input to a posture estimation model, the skeleton of the visible parts is detected by machine learning. Then, skeletal frame information including the positions of the 10 parts that are not visible (right shoulder, right hip, left front ankle, left front hoof, left hind ankle, left hind hoof, right front ankle, right front hoof, right hind ankle, right hind hoof) is output as the posture estimation result. Note that the parts of the cow individual shown here are examples, and the types and numbers of parts included in the image and parts not included in the image are not limited to these and can be set appropriately.

[0087] As a result, even if an image contains an obstruction and only a portion of the individual is included, the entire body region can be identified and used for behavior analysis of the individual.In this way, information on the body parts included in the image is used as information for estimating the body parts not included in the image.In addition, as input data, for example, the positions of each body part detected in images before and after the target image in the time series may be used.

[0088] In this way, the scope of the present invention also includes a configuration in which, in a learning data generation device, a body region identification unit identifies a body region representing the entire body of an individual by referring to the posture of the individual output by inputting image data of the detected moving object as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture of the individual. In addition, such a body region identification unit can also be used in an individual identification device according to one aspect of the present invention.

[0089] [Behavioral Analysis System] The behavioral type (behavioral type) of each individual animal is important information for managing the animal. For example, feeding behavior and resting behavior are indicators that maintenance behavior is ensured. In addition, in ruminant livestock, rumination behavior, among other behavioral types, is considered to be useful information that serves as a barometer reflecting feed intake, signs of various diseases, recovery state, or stress state. Conventionally, systems that grasp behavioral types using wearable sensors and the like have been put into practical use, but there are problems in terms of cost and labor involved in management. On the other hand, conventional image analysis has been difficult to detect behaviors that require grasping of movements, such as rumination. According to one embodiment of the present invention, by grasping the behavioral type of an animal using image analysis, behavioral type data can be obtained easily and labor-savingly. Furthermore, according to one embodiment of the present invention, even behavioral types that require grasping of movements can be detected.

[0090] A behavior-based analysis system according to an embodiment of the present invention will be described with reference to Fig. 21. Fig. 21 is a block diagram showing a configuration of a main part of a behavior-based analysis system 200 according to an embodiment of the present invention. The behavior-based analysis system 200 includes a behavior-based analysis device 220. The behavior-based analysis system 200 also includes a camera 20, a learning data generation device 210, and a learning device 100.

[0091] (Behavioral analysis device 220) The behavioral analysis device 220 includes an image acquisition unit 221, a detection unit 222, a classification unit 223, a position identification unit 224, an information acquisition unit 225, and a behavioral analysis unit 226. The behavioral analysis device 220 further includes an analysis result storage unit 227. The behavioral analysis device 220 detects and analyzes the behavioral patterns of individuals.

[0092] <Individual> The individual whose behavior is analyzed by the behavioral analysis device 220 may be a human, an animal, another moving object, or the like, but may also be an individual managed in a group in a closed space, or an individual managed individually. Such individuals include livestock individuals raised in multiple numbers and livestock individuals raised alone. The livestock individual may be a ruminant individual such as a cow individual, for example, a dairy cow individual, more specifically, a Holstein dairy cow individual. The individual raised in multiple numbers may be an individual allowed to roam freely indoors, or an individual tethered by a chain or the like. The individual raised in multiple numbers may also be an individual raised in an outdoor management area. In this embodiment, a case in which the individual is a cow individual will be described as an example.

[0093] <Image acquisition unit 221> The image acquisition unit 221 acquires an image including an individual cow photographed by the camera 20. The image acquisition unit 221 is configured similarly to the image acquisition unit 121 of the individual identification device 120 described above. Therefore, the description of the image acquisition unit 121 of the individual identification device 120 will be used to explain the image acquisition unit 221, and a detailed description thereof will be omitted. The camera 20 is also the same as the camera 20 described in relation to the individual identification device 120, and a detailed description thereof will be omitted.

[0094] <Detection unit 222> The detection unit 222 detects individual cattle in the image acquired by the image acquisition unit 221. Methods for detecting individual cattle in an image include a moving object detection method and a method using machine learning. When using a moving object detection method, the detection unit 222 can be configured in the same manner as the moving object detection unit 12 of the learning data generation device 10. When using machine learning, a machine learning model such as the object detection machine learning model shown in FIG. 17 and the skeleton detection machine learning model shown in FIG. 18 can be used. Note that when an individual cattle is included alone in an image, it is not necessary to detect the individual cattle, and position identification, which will be described later, may be performed on the acquired image.

[0095] <Identification unit 223> The identification unit 223 identifies the individual cow detected by the detection unit 222. The identification unit 223 is configured similarly to the identification unit 122 of the individual identification device 120 described above. Therefore, the description of the identification unit 122 of the individual identification device 120 will be used to explain the identification unit 223, and a detailed description thereof will be omitted. Note that when a single individual cow is included in an image, it is not necessary to identify the individual cow, and position identification, which will be described later, may be performed on the acquired image.

[0096] <Location specifying unit 224> The position specifying unit 224 specifies the position of the body part of the identified individual. The positions of the body parts of the individual specified by the position specifying unit 224 are, for example, the nose tip, mouth, forelegs, and forelegs. FIG. 22 shows an example of the position specifying unit 224 specifying the body parts of a bovine individual. The left side of FIG. 22 shows an example of specifying the nose tip, mouth, forelegs, and forelegs of a bovine individual in a sitting position, and the right side shows an example of specifying the nose tip, mouth, forelegs, and forelegs of a bovine individual in a standing position. Note that the number of body parts of the individual to be specified is not limited to this, and may be less or more.

[0097] The position identification unit 224 may identify the position of the body part of the individual by using machine learning. For example, an image including an individual is used as input data, and an image acquired by the image acquisition unit is input as input data to a position identification model that has been machine-learned to identify the position of the body part of the individual in the image, thereby identifying the output position. By inputting an image including an individual to the position identification model, an image in which the body part of the individual is identified can be output as shown in FIG. 22.

[0098] The position identifying unit 224 may identify the position by acquiring the coordinates of the position of the body part of the individual. The position identifying unit 224 acquires the position coordinates of the body part of the individual in the image. The position identifying unit 224 may convert the acquired position coordinates into spatial coordinates of the space in which the individual cow exists. The position identifying unit 224 acquires position coordinates in a plurality of frames of images captured at different times.

[0099] The position identification unit 224 may identify the position by referring to the posture of the individual output by inputting the detected image data of the individual as input data to a posture estimation model that has been machine-learned to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture. The posture estimation model used in the position identification unit 224 is the same as the posture estimation model shown in FIG. 20. The position identification unit 224 may also use a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a part of the individual's body used in the enclosing unit 13 of the learning data generation device 10. This makes it possible to identify the part of the individual's body and also to obtain position coordinates even when there is an obstruction in the image and only a part of the individual's body is included.

[0100] In addition, when the image data includes only a part of the individual's body, the position identification unit 224 may identify the body part of the individual in the image of the part of the individual's body and acquire the position coordinates without generating an image representing the entire body of the individual. In this case, for example, the body part of the individual may be identified using a position identification model that has been trained to be able to identify the body part of the individual from the image of the part of the individual's body. Even if the image of the part of the individual's body is used as it is without restoring the image of the entire body from the image of the part of the individual's body, the behavioral analysis can be performed.

[0101] <Information acquisition unit 225> The information acquiring unit 225 acquires time series information representing changes in the identified positions within a predetermined time. The time series information acquired by the information acquiring unit 225 is information representing changes in the parts of the individual cow obtained from successive images within a predetermined time, and may be an average distance between each part and an average displacement between image frames of each part obtained based on the position coordinates acquired by the position identifying unit 224.

[0102] <Behavioral Analysis Department 226> The behavior pattern analysis unit 226 refers to the acquired time-series information to analyze the behavior pattern of the individual cow. By referring to the time-series information, the behavior pattern analysis unit 226 can detect the movement of the individual cow and can also track the movement of the individual cow. This makes it possible to analyze the behavior pattern from the movement and behavior of the individual cow at a specific timing or within a specific time period. The behavior pattern analysis unit 226 stores the analysis result of the behavior pattern of the individual cow in the analysis result storage unit 217.

[0103] The behavioral patterns of individual cattle analyzed by the behavioral pattern analysis unit 226 are types of behavior, such as lying down resting, lying down ruminating, standing upright resting, standing upright ruminating, eating, etc. As shown in Fig. 23, by acquiring in advance information that associates time-series information obtained from successive images within a predetermined time with the behavioral patterns of individual cattle corresponding to the time-series information, it is possible to analyze the behavioral patterns of individual cattle based on the time-series information.

[0104] The behavioral pattern analysis unit 226 may analyze the behavioral pattern of an individual cow using an analysis model constructed using teacher data in which time-series information and the behavioral pattern of the individual cow are associated with each other. The behavioral pattern of the individual cow is output by inputting the time-series information as input data into such an analysis model. The behavioral pattern analysis unit 226 may display the analyzed behavioral pattern of the individual cow on a display unit (not shown) together with an image of the individual cow, as shown in FIG. 23. In addition, when the image data includes only a part of the body of the individual cow, the behavioral pattern analysis unit 226 may analyze the behavioral pattern of the individual cow using only the information on the position coordinates of the specified part of the body. In this case, the behavioral pattern of the individual cow may be analyzed using an analysis model that has been trained so that the behavioral pattern can be analyzed even if there is missing data in the position coordinates of the body part.

[0105] <Learning Data Generating Device 210> The learning data generation device 210 included in the behavioral analysis system 200 includes an image acquisition unit 11 that acquires an image including an individual, a detection unit 212 that detects the individual in the acquired image, an identification unit 213 that identifies a body region of the detected individual, an extraction unit 14 that extracts a body region image of the identified body region, and a learning data generation unit 15 that generates learning data for a discrimination model that identifies individuals from the extracted body region image. The learning data generation device 210 further includes a learning data storage unit 16 that stores the generated learning data.

[0106] The detection unit 212 detects an individual in an image using a moving object detection method, a method using machine learning, etc. Methods for detecting an individual in an image using machine learning include a method for detecting an object using machine learning and a method for detecting the skeleton of an individual. The identification unit 213 identifies the body region of the detected individual using a method for adding a frame to a body region including the entire body of the detected individual, a method using image segmentation, etc.

[0107] The image acquisition unit 11, the extraction unit 14, and the learning data generation unit 15 are the same as those components in the learning data generation device 10 shown in Fig. 1. That is, the learning data generation device 210 is configured similarly to the learning data generation device 10. Therefore, the description of the learning data generation device 10 will be used to describe the learning data generation device 210, and a detailed description thereof will be omitted.

[0108] The training data generation device 210 can be used not only to generate training data for constructing a discrimination model, but also to generate training data for constructing an analysis model that the behavioral type analysis unit 226 uses to analyze behavioral types.

[0109] <Learning device 100> The learning device 100 constructs a discrimination model, an analysis model, and the like, using the learning data generated by the learning data generation device 210. The learning device 100 provided in the behavior-based analysis system 200 is the same as the learning device 100 shown in Fig. 10. Therefore, the description of the learning device 100 provided in the behavior-based analysis system 200 will be given by quoting the description of the learning device 100 shown in Fig. 10, and a detailed description thereof will be omitted.

[0110] <Behavioral analysis processing> An example of behavior-type analysis processing by the behavior-type analysis device 220 will be described with reference to Fig. 24. Fig. 24 is a flowchart showing the behavior-type analysis processing by the behavior-type analysis device 220 of Fig. 23.

[0111] First, in step S241, the image acquisition unit 221 acquires an image including an individual cow. Note that the behavioral analysis process by the behavioral analysis device 220 may be started by accepting input of captured image data from the camera 20, or by accepting an operation input from a user to start the behavioral analysis process.

[0112] Next, the detection unit 222 inputs the image data to a skeleton detection machine learning model and detects individual cattle in the image data (step S242). Then, the identification unit 223 inputs the image data of the detected individual cattle to an identification model, obtains individual identification information of the individual cattle, and identifies the individual cattle (step S243).

[0113] The position identification unit 224 inputs the image data of the identified cow into the position identification model and acquires position coordinates that identify the body parts of the cow (step S244).Then, the information acquisition unit 225 acquires the identified position coordinates for several consecutive frames and calculates the average distance between each part and the average displacement between frames of each part as time-series information (step S245).

[0114] The behavior pattern analysis unit 226 inputs the time-series information into the analysis model and analyzes the behavior pattern of the individual cow (step S246). The behavior pattern analysis unit 226 stores the analysis result in the analysis result storage unit 227 and ends the behavior pattern analysis process.

[0115] In this way, the behavioral analysis device 220 can analyze the behavioral patterns of individuals by identifying the positions of body parts of the individual from the image data and tracking changes in the positions. This makes it possible to perform behavioral analysis with less effort and easily, and can also detect behaviors that require the understanding of actions such as rumination.

[0116] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0117] 10 learning data generation device, 11 image acquisition unit, 12 moving object detection unit, 13 framing unit, 14 extraction unit, 15 learning data generation unit (generation unit), 100 learning device, 101 learning data acquisition unit, 102 learning unit, 120 individual identification device, 121 image acquisition unit, 122 identification unit, 140 behavior analysis device, 141 Identification image acquisition unit, 142 Analysis unit, 200 Behavioral analysis system, 220 Behavioral analysis device, 221 Image acquisition unit, 222 Detection unit, 223 Identification unit, 224 Position identification unit, 225 Information acquisition unit, 226 Behavioral analysis unit

Claims

1. an image acquisition unit for acquiring an image including an individual; A detection unit that detects the individual in the image; a position identification unit that identifies a position of the detected body part of the individual; an information acquisition unit that acquires time-series information representing a change in the identified position within a predetermined time; a behavior pattern analysis unit that refers to the acquired time-series information and analyzes the behavior pattern of the individual; A behavioral analysis device comprising:

2. The behavioral analysis device according to claim 1 , wherein the behavioral type of the individual analyzed by the behavioral analysis unit is selected from the group consisting of recumbent resting, recumbent rumination, standing resting, standing rumination, and feeding.

3. 3. The behavioral analysis device according to claim 1, wherein the position identification unit is configured to identify the position output by inputting the image acquired by the image acquisition unit as input data into a position identification model that has been machine-trained to identify a position of a body part of the individual in an image, using an image including the individual as input data.

4. The behavioral analysis device according to claim 1 , wherein the position specifying unit acquires coordinates of a position of a body part of the individual.

5. 5. The behavioral type analysis device according to claim 1, wherein the behavioral type analysis unit inputs time-series information acquired by the information acquisition unit as input data into an analysis model that has been machine-learned to estimate a behavioral type corresponding to the time-series information, and acquires, as an analysis result, a behavioral type that is output by inputting the time-series information acquired by the information acquisition unit as input data.

6. 6. The behavioral analysis device according to claim 1, wherein the position identification unit identifies the position by referring to the posture of the individual that is output by inputting image data of the individual as input data into a posture estimation model that has been machine-learned to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture of the individual.

7. an identification unit that identifies the individual detected by the detection unit; The behavioral analysis device according to claim 1 , further comprising:

8. An image acquisition step of acquiring an image including an individual; a detection step of detecting the individual in the image; a location step of determining a location of the detected body part of the individual; an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined time; a behavior pattern analysis step of analyzing the behavior pattern of the individual by referring to the acquired time series information; A behavioral analysis method comprising:

9. A behavioral analysis program for causing a computer to function as the behavioral analysis device described in any one of claims 1 to 7, comprising the image acquisition unit, the detection unit, the position identification unit, the information acquisition unit, and the behavioral analysis unit.

10. A computer-readable recording medium having recorded thereon the behavioral analysis program according to claim 9.

Citation Information

Patent Citations

  • Image processing apparatus, image processing method and image processing program

    JP2019003565A

  • Aquatic animal cartilage extracts

    JP2021014463A

  • Recognition data transmission device

    WO2015093330A1

  • Suspicious person detection system

    JP2017224249A

  • Video recording device, video recording method and program

    JP2019169843A