Behavior tracking system, behavior tracking method, and program

Abstract

To provide a technique enabling behavior of an individual of an animal to be tracked with excellent accuracy even when a plurality of individuals of animals is included in a continuous image.SOLUTION: A behavior tracking system (100) comprises: an acquisition unit (21) which acquires a continuous image (400) captured by using an imaging device (10); a division unit (22) which divides the continuous image (400) into individual images (500) along a temporal axis; a detection unit (23) which detects a frame (F) surrounding an individual (200) of an animal and a skeleton (B) of the individual (200) of the animal in the individual images (500); and a tracking unit (24) which tracks the behavior of the individual (200) of the animal on the basis of the information on the temporal axis of the separate image (500), the information on the frame (F) and the information on the skeleton (B) among the plurality of individual images (500).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a behavior tracking system, a behavior tracking method, and a program. [Background technology]

[0002] Tracking the behavior of individual animals is very important in the livestock industry, zoo industry, pet industry, and the like, where health management of animals is required. In recent years, in these industries, a technique for tracking the behavior of individual animals by processing a series of images including animals using an image processing technique has been used.

[0003] As a technology for tracking the behavior of individual animals, a method is known in which a Motion History Image (MHI) is obtained from images captured by a camera installed above a livestock barn, and the riding behavior of cows is interpreted based on the MHI (see, for example, Patent Document 1).

[0004] Furthermore, as a technology for tracking the behavior of individual animals, a monitoring device is known that calculates the amount of movement of an object from multiple captured images including the object, and notifies the user about signs of parturition in the object based on the calculated amount of movement (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 155856 [Patent Document 2] International Publication No. 2017 / 158698 Summary of the Invention [Problem to be solved by the invention]

[0006] In the livestock industry, zoo industry, pet industry, and the like, multiple animals are sometimes kept in one area. When capturing an image of an animal in such an environment, the image may contain multiple animals. Therefore, when applying the above-mentioned conventional technology to such an environment, in order to track the behavior of an individual animal, it is necessary to identify the individual animal from the multiple animals contained in the image in parallel.

[0007] However, in sequential images, the accuracy of individual identification can decrease significantly depending on changes over time in the angle at which individual animals are captured and the distance between individual animals, and as a result, the accuracy of behavior tracking can also decrease significantly.

[0008] An object of one aspect of the present invention is to provide a technique that can track the behavior of individual animals with a high degree of accuracy even when multiple individual animals are included in successive images. [Means for solving the problem]

[0009] In order to solve the above problems, aspects of the present invention include the following aspects.

[0010] (1) A behavior tracking system comprising: an acquisition unit that acquires successive images captured using an image sensor; a division unit that divides the successive images into individual images along a time axis; a detection unit that detects a frame surrounding an individual animal and a skeleton of the individual animal in each of the individual images; and a tracking unit that tracks the behavior of the individual animal among a plurality of the individual images based on information on the time axis, information on the frame, and information on the skeleton of the individual images.

[0011] (2) The behavior tracking system described in (1), wherein in the tracking unit, at least one of the individual images includes a plurality of individual animals.

[0012] (3) The behavior tracking system according to (1) or (2), wherein in the tracking unit, the individual animals contained in the multiple individual images are constant.

[0013] (4) A behavior tracking system described in any of (1) to (3), wherein in the tracking unit, the information of the frame includes information of the coordinates of the frame in the individual image, and the information of the skeleton includes information of the coordinates of the skeleton in the individual image and information of the coordinates of the skeleton in the frame.

[0014] (5) A behavior tracking system described in any of (1) to (4), wherein the detection unit includes a frame detection unit that detects the frame surrounding the individual animal in the individual image, and a skeleton detection unit that detects the skeleton of the individual animal within the frame detected by the frame detection unit.

[0015] (6) The behavior tracking system according to any one of (1) to (5), wherein the frame has a rectangular shape.

[0016] (7) The behavior tracking system according to any one of (1) to (6), further comprising an identification unit that identifies an individual animal in at least one of the individual images.

[0017] (8) The behavior tracking system according to any one of (1) to (7), wherein the animal is a livestock animal.

[0018] (9) A behavior tracking method including: an acquisition step of acquiring successive images captured using an imaging element; a division step of dividing the successive images into individual images along a time axis; a detection step of detecting a frame surrounding an individual animal and a skeleton of the individual animal in each of the individual images; and a tracking step of tracking behavior of the individual animal among a plurality of the individual images based on information on the time axis, information on the frame, and information on the skeleton of the individual images.

[0019] (10) A behavior tracking program for causing a computer to function as the behavior tracking system described in any one of (1) to (8), the program causing a computer to function as the acquisition unit, the division unit, the detection unit, and the tracking unit. Effect of the Invention

[0020] According to one aspect of the present invention, even when multiple individual animals are included in successive images, the behavior of individual animals can be tracked with high accuracy. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a configuration of a main part of a behavior tracking system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an example of an animal rearing style in one embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram illustrating an overview of processing by a division unit in one embodiment of the present invention. [Figure 4] 5A to 5C are diagrams illustrating an overview of processing by a detection unit in the embodiment of the present invention. [Diagram 5] 4 is a diagram for explaining an overview of processing by a tracking unit in one embodiment of the present invention. FIG. [Figure 6] 1 is a flowchart for explaining an overview of a behavior tracking method by a behavior tracking system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] [Features of the behavior tracking system of the present invention] A behavior tracking system according to one embodiment of the present invention includes an acquisition unit that acquires successive images captured using an image sensor, a division unit that divides the successive images into individual images along a time axis, a detection unit that detects a frame surrounding an individual animal and a skeleton of the individual animal in each individual image, and a tracking unit that tracks behavior of the individual animal among a plurality of the individual images based on information on the time axis, information on the frame, and information on the skeleton of the individual images.

[0023] The behavior tracking system according to one aspect of the present invention tracks the behavior of individual animals based on the above-mentioned information, and can therefore track the behavior of individual animals with good accuracy even when multiple individual animals are included in consecutive images. The behavior tracking system according to one aspect of the present invention can track the behavior of individual animals with good accuracy even when there is no information such as teacher data that is input in advance to identify individual animals. Since the behavior tracking system according to one aspect of the present invention can track the behavior of multiple individual animals with good accuracy, it can also be used suitably in a breeding style in which the breeding area per individual animal is small and multiple individual animals are likely to be included in consecutive images.

[0024] [Behavior Tracking System] A behavior tracking system according to one aspect of the present invention will be described in detail below. The behavior tracking system according to one embodiment of the present invention is a system that tracks the behavior of an individual animal based on continuous images. In the present invention, the term "tracking the behavior of an individual animal" refers to at least one of storing or displaying information about the state or progress of an action, such as the position, posture, or moving speed, for each individual animal. Therefore, in the present invention, the term "tracking the behavior of an individual animal" does not need to include identifying the individual animal. In addition, in the present invention, the term "continuous images" refers to a group of multiple individual images captured using a specific image sensor at different times. In addition, in the present invention, the term "individual images" refers to still images captured at a specific time.

[0025] With reference to FIG. 1, a behavior tracking system according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing a main configuration of the behavior tracking system according to an embodiment of the present invention. As shown in FIG. 1, the behavior tracking system 100 includes an image sensor 10, a behavior tracking device 20, and a user terminal 30. The image sensor 10, the behavior tracking device 20, and the user terminal 30 may be integrated into a terminal such as a camera, or may be connected to each other via a network or a server. The behavior tracking device 20 includes an acquisition unit 21, a division unit 22, a detection unit 23, a tracking unit 24, an identification unit 25, a storage unit 26, a display unit 27, an analysis unit 28, and a control unit (not shown). The behavior tracking device 20 may be implemented in the form of an integrated circuit such as an IC chip, or in the form of a calculator such as a computer equipped with a processor and a memory.

[0026] (Behavior tracking target) The target of behavior tracking by the behavior tracking system 100 is an animal. Examples of animals include animals that can be used for industrial purposes, such as livestock animals, zoo animals, and pet animals. Tracking the behavior of an individual animal that can be used for industrial purposes is extremely important for understanding the health condition of the individual and for raising and breeding the animal. Examples of livestock animals include cows, pigs, sheep, goats, horses, chickens, mice, rats, and guinea pigs. Examples of zoo animals include lions, tigers, monkeys, bears, elephants, and giraffes. Examples of pet animals include dogs, cats, and rabbits. Since the scale of breeding for industrial purposes is large, it is preferable for the animal to be a livestock animal, and more preferably a cow.

[0027] In one embodiment of the present invention, the manner in which the animals are kept is not limited as long as the imaging element 10 can capture an image of the animals. Examples of the manner in which the animals are kept include a free-range manner in which the animals are kept free in a divided management area, and a tethered manner in which the animals are kept tied with a chain or the like indoors or outdoors. The number of animals kept is one or more, and may be one or more. The type of animals kept may be one type or multiple types, and is preferably one type. By keeping only one type of animal, the behavior of each individual animal can be tracked with better accuracy.

[0028] An example of the animal rearing style will be described with reference to FIG. 2. FIG. 2 is a diagram for explaining an example of the animal rearing style in one embodiment of the present invention. The rearing style shown in FIG. 2 is a free-range style in which a plurality of animal individuals 200 are allowed to roam freely in an outdoor management area 300. As shown in FIG. 2, a plurality of image sensors 10 are installed so that at least a part of the management area 300 is included within the angle of view of the image sensor 10. In such a rearing style, a plurality of animal individuals 200 may be included in a continuous image captured by the image sensor 10. Furthermore, in such a rearing style, a plurality of animal individuals 200 may move simultaneously in the continuous image captured by the image sensor 10, a plurality of animal individuals 200 may overlap each other, and a combination of these may occur. The behavior tracking system 100 according to one embodiment of the present invention is provided with the tracking unit 24 having the above-mentioned configuration, and therefore, even in such a rearing style, the behavior of the individual animal can be tracked with good accuracy.

[0029] (Image sensor) The imaging element 10 captures successive images of an animal that is the subject of behavior tracking. Information on the successive images captured by the imaging element 10 is sent to the acquisition unit 21 by the control unit. Hereinafter, when the term "successive images" is simply used, it is intended to mean "information on the successive images." Furthermore, the "information" may be implemented in the form of data or a collection of data.

[0030] In one embodiment of the present invention, the imaging element 10 is not limited and may be a conventionally known imaging element as long as it can capture an image of an animal in continuous images. Examples of the imaging element 10 include imaging elements such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor, as well as cameras equipped with imaging elements. Examples of cameras equipped with imaging elements include video cameras and time lapse cameras. Examples of cameras equipped with imaging elements include visible light cameras, infrared cameras, and combinations thereof. In the present invention, the term "visible light camera" refers to a camera equipped with an optical system such as a lens, a light detection element, and the like, which detects light from a subject and generates continuous visible light images. In the present invention, the term "infrared camera" refers to a camera equipped with an optical system such as a lens, an infrared detection element, and the like, which detects infrared rays emitted from a subject and generates continuous infrared images. In the present invention, the term "continuous infrared images" refers to continuous images in which the temperature of each part of the subject is represented by color.

[0031] The behavior tracking system 100 preferably includes a plurality of image capturing elements 10. This allows a plurality of consecutive images captured from various directions to be obtained. By tracking the behavior of the individual animal 200 for each of these consecutive images, the behavior of the individual animal 200 can be tracked with higher accuracy. There is no particular upper limit on the number of image capturing elements 10.

[0032] The behavior tracking system 100 preferably includes at least one visible light camera and at least one infrared camera as the image sensor 10. This makes it possible to capture images of animals both day and night, for example, by capturing images of the animals with the visible light camera during the day and the infrared camera at night.

[0033] There are no limitations on the installation position and orientation of the imaging element 10 as long as the imaging element 10 can capture an image of an animal. The installation position and orientation of the imaging element 10 are preferably such that they do not interfere with the animal's behavior and provide a wide imaging range, and more preferably such that they enable the animal to be imaged from diagonally above.

[0034] In one embodiment of the present invention, the image sensor 10 is controlled by a control unit (not shown) included in the behavior tracking device 20. The image sensor 10 may be automatically controlled based on preset parameters, or may be manually controlled based on input via the user terminal 30. In controlling the image sensor 10, imaging conditions such as imaging duration, imaging interval, direction, focus, and zoom can be appropriately adjusted by the user according to the type of animal, the number of individual animals 200, the average moving speed of the animal, and the breeding style. For example, the imaging interval of consecutive images, i.e., the ratio between the frame rate (1 / sec) and the average moving speed (m / sec) of the animal, may be appropriately adjusted. By appropriately adjusting the ratio, the behavior of the individual animals 200 can be tracked with better accuracy.

[0035] (Control unit) The control unit (not shown) controls the acquisition unit 21, division unit 22, detection unit 23, tracking unit 24, identification unit 25, storage unit 26, display unit 27, and analysis unit 28 included in the behavior tracking device 20. The control unit can be implemented in the form of a combination of a storage medium such as a memory and a processor that executes a program stored in the storage medium.

[0036] (Acquisition Department) The acquisition unit 21 acquires continuous images captured using the imaging element 10. The acquisition unit 21 may be implemented on a non-volatile memory such as a hard disk drive (HDD) and a read only memory (ROM). The processing performed by the acquisition unit 21 may be controlled by a processor included in a control unit (not shown).

[0037] The acquisition unit 21 may acquire a plurality of consecutive images captured using a plurality of image sensors 10. The acquisition unit 21 may acquire consecutive images each time the image sensor 10 captures consecutive images, or may acquire consecutive images at predetermined time intervals. The acquisition unit 21 sends the acquired consecutive images to the division unit 22. The acquisition unit 21 may also send the acquired consecutive images to at least one of the storage unit 26 and the display unit 27.

[0038] (divided part) The division unit 22 divides the continuous images acquired by the acquisition unit 21 into individual images along a time axis. The division unit 22 may be implemented on a non-volatile memory such as a hard disk drive (HDD) and a read only memory (ROM). The processing performed by the division unit 22 may be controlled by a processor included in a control unit (not shown).

[0039] The dividing unit 22 included in the behavior tracking system 100 will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining an overview of the processing of the dividing unit 22 in one embodiment of the present invention. The dividing unit 22 divides the continuous images 400 into individual images 500 included in the continuous images 400 for each time when the individual images 500 were captured, and generates the individual images 500. The information of each individual image 500 includes information on the time axis in addition to information on each pixel constituting a still image. The dividing unit 22 sends information on the generated individual images 500 to the detection unit 23. The dividing unit 22 may also send information on the generated individual images 500 to at least one of the tracking unit 24, the identification unit 25, the storage unit 26, the display unit 27, and the analysis unit 28. Hereinafter, when the term "individual images" is simply used, it is intended to mean "information on the individual images".

[0040] The individual images 500 sent by the division unit 22 to at least one of the detection unit 23, the tracking unit 24, the identification unit 25, the storage unit 26, the display unit 27, and the analysis unit 28 may be all or a part of the generated individual images 500. The division unit 22 may, for example, select some of the generated individual images 500 and send them to the detection unit 23, the tracking unit 24, the identification unit 25, the storage unit 26, the display unit 27, and the analysis unit 28. A method for selecting some individual images 500 may be a conventionally known method. As a method for the division unit 22 to select some individual images 500, for example, a method of selecting individual images 500 at intervals of a preset number or time intervals from all the generated individual images 500 by referring to information on the time axis, etc., may be mentioned. The individual images 500 sent to each of the detection unit 23, tracking unit 24, identification unit 25, memory unit 26, display unit 27, and analysis unit 28 may be different sets or the same set, and preferably are the same set.

[0041] (Detection unit) The detection unit 23 detects a frame surrounding the individual animal 200 and a skeleton of the individual animal 200 in each image 500. The detection unit 23 may be implemented on a non-volatile memory such as a hard disk drive (HDD) and a read only memory (ROM). The processing performed by the detection unit 23 may be controlled by a processor included in a control unit (not shown).

[0042] The detection unit 23 included in the behavior tracking system 100 will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining an overview of the processing of the detection unit 23 in one embodiment of the present invention. The detection unit 23 detects a frame F surrounding the individual animal 200 and a skeleton B of the individual animal 200 for each of the individual images 500 divided by the division unit 22. Next, the detection unit 23 associates the frame F surrounding the individual animal 200 with the skeleton B of the individual animal 200 surrounded by the frame. Next, the detection unit 23 sends information on the result of the processing to the tracking unit 24. The detection unit 23 may also send information on the result of the processing to at least one of the storage unit 26 and the display unit 27.

[0043] The detection unit 23 detects a frame F surrounding the individual animal 200 in the individual image 500. In the present invention, the term "detecting a frame" is intended to obtain information on the coordinates of pixels constituting a line surrounding a specific area in the individual image 500. The frame F may surround the entire body of the individual animal 200, or may surround only a part of the individual animal 200. It is preferable that the frame F surrounds a part of the entire body of the individual animal 200 that is included in the individual image 500. That is, when the individual image 500 includes the entire body of the individual animal 200 in a still image, it is preferable that the frame F surrounds the entire body of the individual animal 200. Also, when the individual image 500 includes only a part of the individual animal 200 in a still image, it is preferable that the frame F surrounds all of the part of the individual animal 200 that is included in the individual image 500 in the still image. This makes the portion of the entire body of the individual animal 200 enclosed by the frame F larger, and by improving the one-to-one correspondence between the information of the frame F and the information of the individual animal 200, it is possible to track the behavior of the individual animal 200 with better accuracy. Also, it is preferable that the frame F is as small as possible as long as it surrounds the portion of the entire body of the individual animal 200 that is included in the individual image 500. This makes the portion of the portion enclosed by the frame F that is not the individual animal 200 smaller, and by improving the one-to-one correspondence between the information of the frame F and the information of the individual animal 200, it is possible to track the behavior of the individual animal 200 with better accuracy.

[0044] The number of frames F detected in an individual image 500 is preferably equal to the number of individual animal 200 included in the individual image 500. In other words, it is preferable that the frames F and the individual animal 200 correspond one-to-one. This increases the one-to-one correspondence between the information of the frames F and the information of the individual animal 200, thereby making it possible to track the behavior of the individual animal 200 with higher accuracy. Furthermore, when multiple frames F are detected in an individual image 500, the multiple frames F may overlap each other. When multiple frames F overlap each other, the multiple frames F may completely overlap each other or may only partially overlap each other.

[0045] The shape of the frame F can be appropriately adjusted by the user according to the individual animal 200 surrounded by the frame F. Examples of the shape of the frame F include a rectangle, a square, a rectangle, a trapezoid, a parallelogram, a rhombus, a circle, and an ellipse, as well as a shape that follows the contour of the individual animal 200 surrounded by the frame F. In addition, the shape of the frame F is preferably a rectangle. When the shape of the frame F is a rectangle, information of the frame F, such as the area of ​​the region surrounded by the rectangle and the center of gravity of the rectangle, can be easily calculated based on the coordinates of each vertex of the rectangle in the individual image 500. Therefore, when the shape of the frame F is a rectangle, the tracking unit 24 can easily handle the information when comparing the information of the frame F.

[0046] The method of detecting the frame F surrounding the individual animal 200 may be a conventionally known method. For example, the method of detecting the frame F may be a method of using an object detection model with an individual image 500 as an input. The object detection model may be a conventionally known object detection model, or may be an object detection model that has been independently developed. In addition, the conventionally known object detection model may be a trained object detection model that has been released in a state in which training data has been trained in advance, or may be a retrained object detection model in which independently generated training data has been retrained for the trained object detection model. For example, conventionally known object detection models include Yolo (You Only Look Once), Faster R-CNN (Convolutional Neural Network), SSD (Single Shot Multibox Detector), and DEtection TRansformer (DETR).

[0047] The detection unit 23 detects the skeleton B of the individual animal 200 in the individual image 500. The skeleton B may be any skeleton of the individual animal 200 that can be detected from the individual image 500, which is a still image. Here, the term "skeleton of the individual animal 200" refers to a part that can be uniquely determined as a feature point on the surface or inside of the body of the individual animal 200. Examples of the skeleton B include bones, joints, parts, tips of parts, and combinations thereof. Examples of the bones include the skull, cervical vertebrae, thoracic vertebrae, lumbar vertebrae, sacral vertebrae, coccygeal vertebrae, pelvis, ribs, sternum, scapula, humerus, ulna, radius, carpal bones, metacarpal bones, phalanges, ligaments, femurs, patellas, tibias, tarsal bones, metatarsals, and phalanges. Examples of the joints include the atlanto-occipital joint, costovertebral joint, articular joint, sacroiliac joint, sternocostal joint, costochondral joint, shoulder joint, elbow joint, carpal joint, metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, hip joint, femoropatellar joint, femoro-tibial joint, tarsal joint, metatarsophalangeal joint, proximal interphalangeal joint, and distal interphalangeal joint. Examples of the parts include the head, eyes, nose, mouth, ears, horns, feet, hooves, abdomen, back, and tail. Skeleton B may be one bone, joint, part, or tip of a part, or may be multiple bones, joints, parts, or tips of parts, or may be a combination of these.

[0048] The method for detecting the skeleton B of the individual animal 200 may be a conventionally known method. Examples of the method for detecting the skeleton B include a method of using an individual image 500 as an input and using a skeleton detection model. The skeleton detection model may be a conventionally known skeleton detection model, or may be a uniquely developed skeleton detection model. In addition, the conventionally known skeleton detection model may be a trained skeleton detection model that is released in a state in which training data has been trained in advance, or may be a retrained skeleton detection model in which training data uniquely generated for a trained object detection model is retrained. Examples of conventionally known skeleton detection models include DeepLabCut, OpenPose, DeepPose, and DeeperCUT.

[0049] The detection unit 23 may detect the frame F and the skeleton B independently, or may perform the remaining one by referring to the result of one. Preferably, the detection unit 23 includes a frame detection unit that detects the frame F surrounding the individual animal 200 in the individual image 500, and a skeleton detection unit that detects the skeleton B of the individual animal 200 within the frame F detected by the frame detection unit. That is, the detection unit 23 may include a frame detection unit that first detects the frame F in the individual image 500, and a skeleton detection unit that then detects the skeleton B within the frame F. As a result, the detection unit 23 detects the skeleton B within the frame F that is the same as or smaller than the entire individual image 500, so that the skeleton B can be detected with good accuracy. Also, as a result, the detection unit 23 can detect the skeleton B by referring to the information of the frame F, so that even if there are multiple types of animals kept, the skeleton B can be detected with good accuracy.

[0050] Next, the detection unit 23 associates information on the frame F surrounding the individual animal 200 with information on the skeleton B of the individual animal 200 surrounded by the frame F. The association between the information on the frame F and the information on the skeleton B is performed based on the information on the frame F and the information on the skeleton B. As an association method, for example, a method of associating the frame F with the skeleton B detected within the area surrounded by the frame F can be given. For example, as shown in FIG. 4, the detection unit 23 assigns a label of "0" to the frame F surrounding the individual animal 200 and the skeleton B of the individual animal 200 surrounded by the frame F, and associates them with each other as the frame F(0) and the skeleton B(0), respectively. Here, the frame F(0) and the skeleton B(0) may correspond to the same individual animal 200.

[0051] Next, the detection unit 23 generates information of the result of the above processing by the detection unit 23 (hereinafter, may be referred to as "information by the detection unit 23" or the like). The information by the detection unit 23 includes: information on the frame F; information on the skeleton B; information on the association between the information on the frame F and the information on the skeleton B; and information on the time axis included in the individual image 500 in which the frame F and the skeleton B are detected. Note that the information on the frame F; the information on the skeleton B; information on the association between the information on the frame F and the information on the skeleton B; and information on the time axis included in the individual image 500 in which the frame F and the skeleton B are detected may be set by the user according to the manner of comparison and estimation by the tracking unit 24 described later.

[0052] Next, the detection unit 23 sends the information by the detection unit 23 to the tracking unit 24. The detection unit 23 may also send the information by the detection unit 23 to at least one of the storage unit 26 and the display unit 27.

[0053] (Tracking Department) The tracking unit 24 tracks the behavior of the individual animal 200 among the multiple individual images 500 based on the information on the time axis of the individual images 500, the information on the frame F, and the information on the skeleton B. The tracking unit 24 tracks the behavior of the individual animal 200 among the multiple individual images 500 by referring to the information on the time axis of the individual images 500, the information on the frame F, and the information on the skeleton B. Therefore, the behavior tracking system 100 can track the behavior of the individual animal 200 with good accuracy even when there is no information other than the continuous images 400 captured by the imaging element 10, for example, when there is no information such as teacher data input in advance to identify the individual animal 200. The tracking unit 24 can be implemented on a non-volatile memory such as a hard disk drive (HDD) and a read only memory (ROM). In addition, the processing performed by the tracking unit 24 can be controlled by a processor provided in a control unit (not shown).

[0054] The tracking unit 24 included in the behavior tracking system 100 will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining an overview of the processing of the tracking unit 24 in one embodiment of the present invention. The tracking unit 24 compares information generated by the detection unit 23 from each of the multiple individual images 500 between the multiple individual images 500, and estimates the frame F and skeleton B corresponding to the same animal individual 200 according to the similarity of each piece of information. Next, the tracking unit 24 associates, for each animal individual 200, information on the time axis, and information on the frame F and skeleton B estimated to correspond to the animal individual 200. Next, the tracking unit 24 sends information on the result of the processing to at least one of the storage unit 26 and the display unit 27. The tracking unit 24 may also send information on the result of the processing to the analysis unit 28.

[0055] The tracking unit 24 compares the information generated by the detection unit 23 from each of the multiple individual images 500 between the multiple individual images 500, and estimates the frame F and skeleton B corresponding to the individual 200 of the same animal according to the similarity of each piece of information. Specifically, when there are frames F and skeletons B whose information by the detection unit 23 is similar to each other between the multiple individual images 500, the tracking unit 24 estimates that the group of frames F and skeletons B is the frame F and skeleton B corresponding to the individual 200 of the same animal. Here, whether or not there are frames F and skeletons B whose information by the detection unit 23 is similar to each other is determined depending not only on the similarity of the information of the frame F and skeleton B but also on the information of the time axis. The tracking unit 24 assumes that the frame F and skeleton B corresponding to the individual 200 of the same animal have a greater similarity as the information on the time axis between the multiple individual images 500 is similar. Based on this assumption, the tracking unit 24 weights the similarity of the information of the frame F and the skeleton B more heavily as the information of the time axis of the frame F and the skeleton B is more similar, and weights the similarity of the information of the frame F and the skeleton B less heavily as the information of the time axis of the frame F and the skeleton B is more dissimilar, and determines whether or not there is a frame F and a skeleton B that are similar to each other. The weighting method may be appropriately set by the user. As a weighting method, for example, a method of weighting 1 when the information of the time axis is similar beyond a threshold, and 0 when the information is similar without exceeding the threshold, etc. can be mentioned.

[0056] For example, as shown in FIG. 5, the tracking unit 24 estimates that, among the multiple individual images 500, a frame F(1) and a skeleton B(1) labeled with "1" and a frame F(3) and a skeleton B(3) labeled with "3" are the frame F and the skeleton B corresponding to the individual animal 200. Next, as shown in FIG. 5, for example, the tracking unit 24 associates the information of the frame F and the information of the skeleton B by labeling them as a frame F(X) and a skeleton B(X). Similarly, as shown in FIG. 5, for example, the tracking unit 24 estimates that, among the multiple individual images 500, a frame F(2) and a skeleton B(2) labeled with "2" and a frame F(4) and a skeleton B(4) labeled with "4" are the frame F and the skeleton B corresponding to the individual animal 200. Next, for example, the tracking unit 24 associates the information of the frame F with the information of the skeleton B by labeling them with "Y" and making them a frame F(Y) and a skeleton B(Y) as shown in Fig. 5. Note that in Fig. 5, the frame F and the skeleton B are depicted as being superimposed on an individual image 500 for better understanding of the present invention, but it should be noted that the processing by the tracking unit 24 is substantially performed with reference to the information by the detection unit 23, and the individual image 500 is not necessarily required.

[0057] The time axis information, frame F information, and skeleton B information used by the tracking unit 24 in the comparison and estimation can be appropriately selected by the user.

[0058] Examples of time axis information used by the tracking unit 24 in comparison and estimation include information on the time when the individual image 500 was captured, information on the time required from the start of capturing the consecutive images 400 until the individual image 500 was captured, and information on the order in which the individual image 500 was captured in the group of individual images 500 included in the consecutive images 400.

[0059] Examples of the information on the frame F used by the tracking unit 24 in the comparison and estimation include information on the coordinates of the frame F in the individual image 500, information on the area of ​​the region surrounded by the frame F, and information on the displacement of the frame F between the multiple individual images 500. Examples of the information on the coordinates of the frame F include information on the coordinates of the vertices of the frame F, information on the distance between the vertices of the frame F, information on the coordinates of the center of gravity of the frame F, and information on the coordinates of each point constituting the frame F. Examples of the information on the displacement of the frame F between the multiple individual images 500 include the moving speed of the vertices of the frame F and the moving speed of the center of gravity of the frame F. The type of information on the frame F may be one type or multiple types. Preferably, the information on the frame F includes information on the coordinates of the frame F in the individual images 500. By including information on the coordinates of the frame F in the individual images 500, the tracking unit 24 can capture the change in the information on the frame F accompanying the movement of the individual animal 200 with good accuracy.

[0060] Examples of the information on the skeleton B used by the tracking unit 24 in the comparison and estimation include information on the coordinates of the skeleton B, information on the dimensions of the skeleton B, and information on the displacement of the skeleton B between a plurality of individual images. The information on the coordinates of the skeleton B may be, for example, information on the coordinates of the skeleton B in the individual image 500, or information on the coordinates of the skeleton B in the frame F, or a combination of these. Examples of the information on the displacement of the skeleton B between a plurality of individual images 500 include, for example, the moving speed of the skeleton B. The type of information on the skeleton B may be one type or multiple types. Preferably, the information on the coordinates of the skeleton B includes information on the coordinates of the skeleton B in the individual image 500 and information on the coordinates of the skeleton B in the frame F. By including the information on the coordinates of the skeleton B in the individual image 500 and information on the coordinates of the skeleton B in the frame F, the tracking unit 24 can capture changes in the information on the frame F that do not involve the movement of the individual animal 200 with good accuracy.

[0061] The method used by the tracking unit 24 in the comparison and estimation is not particularly limited as long as it can compare the information on the time axis, the information on the frame F, and the information on the skeleton B. Examples of the method used by the tracking unit 24 in the comparison and estimation include a class classification method, a clustering method, and a combination of these. Examples of the class classification method include a k-NN (k-nearest neighbor) method and a modified k-NN method. Examples of the clustering method include a k-means method and a k-medoids method. Examples of the combination of the class classification and clustering methods include a method in which a clustering method is applied to information generated by the detection unit 23 from some of the multiple individual images 500, and the comparison and estimation are performed, and the formed clusters are used as teacher data to apply a class classification method to information generated by the detection unit 23 from the remaining individual images 500. The tracking unit 24 may appropriately weight each piece of information to perform the comparison and estimation.

[0062] The number of individual images 500 from which the information from the detection unit 23, which the tracking unit 24 performs the comparison and estimation process, is derived is not particularly limited as long as it is plural. As described above, the tracking unit 24 determines whether or not there are frames F and skeletons B that are similar to each other by weighting the similarity between the frames F and skeletons B more heavily as the frames F and skeletons B have more similar time axis information, and weighting the similarity between the frames F and skeletons B less heavily as the frames F and skeletons B have less similar time axis information. Therefore, even if the number of individual images 500 from which the information is derived is three or more, the behavior of the individual animal 200 can be tracked with good accuracy.

[0063] The number of individual animal 200 contained in each of the multiple individual images 500 from which information by the detection unit 23, from which the tracking unit 24 performs the comparison and estimation process, is arbitrary and may be, for example, 0, 1, or more than one. Since the tracking unit 24 performs the comparison and estimation by referring to the time axis information, the frame F information, and the skeleton B information, even if multiple individual animal 200 are included in the individual images 500, it is possible to track the behavior of the individual animal 200 with good accuracy.

[0064] The number of individual animal 200 included in each of the multiple individual images 500 from which the information by the detection unit 23, which the tracking unit 24 processes to compare and estimate, is derived may be constant or may not be constant. Furthermore, the number of individual animal 200 included in each of the multiple individual images 500 from which the information is derived may not be constant. That is, the individual animal 200 may move across the boundary of the angle of view of the image sensor 10 while the image sensor 10 captures the continuous images 400. When there are no frames F and skeletons B with similar information among the multiple individual images 500, the tracking unit 24 estimates that the frames F and skeletons B correspond to different individual animal 200. Therefore, even if the number of individual animal 200 and the number of individual animal 200 included in the multiple individual images 500 from which the information by the detection unit 23, which the tracking unit 24 processes, is not constant, the tracking unit 24 can track the behavior of the individual animal 200 with good accuracy. At least one individual image 500 of the plurality of individual images 500 from which the images are derived preferably includes a plurality of individual animals 200. This allows the behavior of the plurality of individual animals 200 to be tracked simultaneously, and, for example, information regarding the relationship between the behaviors of the plurality of individual animals 200 can be obtained. In addition, the individual animals 200 included in the plurality of individual images 500 from which the images are derived are preferably constant. That is, preferably, the individual animals 200 do not move across the boundary of the angle of view of the image sensor 10 while the image sensor 10 captures the consecutive images 400. This allows the behavior of the individual animals 200 to be tracked with better accuracy.

[0065] Next, the tracking unit 24 associates, for each individual animal 200, time axis information, and information on the frame F and information on the skeleton B that are estimated to correspond to the individual animal 200. Next, the tracking unit 24 generates information resulting from the above processing (hereinafter, may be referred to as "information by the tracking unit 24" or the like). The information by the tracking unit 24 includes: information on the frame F; information on the skeleton B; information on the association between information on the frame F and information on the skeleton B that are estimated to correspond to the same individual animal 200; and information on the time axis included in each individual image 500 in which the frame F and skeleton B are detected.

[0066] Next, the tracking unit 24 sends the information generated by the tracking unit 24 to at least one of the storage unit 26 and the display unit 27. The tracking unit 24 may also send the information generated by the tracking unit 24 to the analysis unit .

[0067] (Analysis Department) The analysis unit 28 analyzes the information by the tracking unit 24 for each individual animal 200. Then, the analysis unit 28 sends the information of the analysis result to at least one of the storage unit 26 and the display unit 27. The method used by the analysis unit 28 in the analysis may be a conventionally known method, and may be appropriately selected by the user according to the desired information regarding the behavior of the individual animal 200. Examples of the method used by the analysis unit 28 in the analysis include a method of detecting a specific behavior of the individual animal 200 by calculating the shape of the frame F and the transition of the position of the skeleton B according to time for each individual animal 200, and a method of determining a specific posture of the individual animal 200. Examples of the specific behavior include walking, sleeping, mounting, and feeding. Examples of the specific posture include standing and lying.

[0068] (Identification section) The identification unit 25 identifies the individual animal 200 in at least one individual image 500. In the present invention, the term "identify" is intended to distinguish each individual animal 200 and to specify information corresponding to each individual animal 200, such as a serial number and a name. The identification unit 25 then sends information on the result of the identification to at least one of the storage unit 26 and the display unit 27. The identification unit 25 may be implemented on a non-volatile memory such as a hard disk drive (HDD) and a read only memory (ROM). Moreover, the processing performed by the identification unit 25 may be controlled by a processor included in a control unit (not shown).

[0069] The method by which the identification unit 25 identifies the individual animal 200 may be a conventionally known method. For example, the method by which the identification unit 25 identifies the individual animal 200 may be a method of identifying the individual animal 200 by using information of the individual animal 200 input in advance as teacher data and information of the individual animal 200 obtained from the individual image 500 as input data. For example, the information of the individual animal 200 input in advance may be information on the skeleton, body type, hair pattern, part shape, posture, behavior, gait, movement line, and place of stay. For example, the method by which the identification unit 25 identifies the individual animal 200 may be a method of identifying the individual animal 200 based on information on a tag attached to the individual animal 200 included in the individual image 500. Examples of the tag include a plate with a serial number written thereon, an IC tag, a radio tag, and an acceleration sensor. These methods may be performed by mechanical calculation such as a computer, or may be performed manually such as by the naked eye. The identification unit 25 may perform the identification and calculate the likelihood of the identification result.

[0070] The classification unit 25 processes one or more individual images 500. For example, the classification unit 25 may process all of the individual images 500 divided by the division unit 22, or may process only the individual images 500 selected by a predetermined method. For example, the classification unit 25 may process a plurality of individual images 500, and send only information on a classification result with a high likelihood among the classification results to at least one of the storage unit 26 and the display unit 27.

[0071] (Storage part) The storage unit 26 stores a program for realizing the functions of the behavior tracking device 20. The storage unit 26 also stores the continuous images 400 acquired by the acquisition unit 21, the individual images 500 divided by the division unit 22, information by the detection unit 23, information by the tracking unit 24, information on the results of analysis by the analysis unit 28, information on the results of identification by the identification unit 25, and the like. The storage unit 26 may be implemented on a non-volatile memory such as an HDD (Hard Disk Drive) and a ROM (Read Only Memory). Writing and reading of the memory of the storage unit 26 may be controlled by a processor provided in a control unit (not shown).

[0072] (Display) The display unit 27 displays the continuous images 400 acquired by the acquisition unit 21, the individual images 500 divided by the division unit 22, information by the detection unit 23, information by the tracking unit 24, information on the results of analysis by the analysis unit 28, information on the results of identification by the identification unit 25, and the like, to the user via the user terminal 30. The display unit 27 may be implemented on a non-volatile memory such as an HDD (Hard Disk Drive) and a ROM (Read Only Memory). Furthermore, the processing performed by the display unit 27 may be controlled by a processor included in a control unit (not shown).

[0073] (User terminal) The user terminal 30 is at least one of a terminal into which a user inputs a signal for controlling the behavior tracking device 20 and a terminal that outputs a signal from the display unit 27 of the behavior tracking device 20 to the user. Examples of the user terminal 30 include a computer, a smartphone, a tablet, a mobile phone, and a camera.

[0074] [Behavior Tracking Method] A behavior tracking method according to one embodiment of the present invention includes an acquisition step of acquiring successive images captured using an imaging element, a division step of dividing the successive images into individual images along a time axis, a detection step of detecting a frame surrounding an individual animal and a skeleton of the individual animal in each of the individual images, and a tracking step of tracking behavior of the individual animal among a plurality of the individual images based on information on the time axis, information on the frame, and information on the skeleton of the individual images.

[0075] As an embodiment of the behavior tracking method according to an aspect of the present invention, a behavior tracking method using the above-mentioned behavior tracking system 100 will be described with reference to Fig. 6. Fig. 6 is a flowchart outlining the behavior tracking method by the behavior tracking system 100 according to one embodiment of the present invention. Therefore, the description of the behavior tracking method according to an aspect of the present invention follows the description of the above-mentioned behavior tracking system 100. The behavior tracking method described below is a method controlled by a control unit (not shown) of the behavior tracking device 20.

[0076] 6, first, the consecutive images 400 captured using the imaging element 10 are acquired (step S1). In step S1, the acquisition unit 21 acquires the consecutive images 400 from the imaging element 10.

[0077] Next, the continuous images 400 acquired in step 1 are divided into individual images 500 along the time axis (step S2). In step S2, the division unit 22 divides the continuous images 400 into individual images 500 along the time axis.

[0078] Next, in each individual image 500 divided in step S2, a frame F surrounding the individual animal 200 and a skeleton B of the individual animal 200 are detected (step S3). In step S3, the detection unit 23 detects the frame F surrounding the individual animal 200 and the skeleton B of the individual animal 200.

[0079] Next, among the multiple individual images 500 including the information of the frame F and the skeleton B detected in step S3, the behavior of the individual animal 200 is tracked based on the information on the time axis of the individual images 500, the information on the frame F, and the information on the skeleton B (step S4). In step S4, the tracking unit 24 tracks the behavior of the individual animal 200 among the multiple individual images 500 based on the information on the time axis of the individual images 500, the information on the frame F, and the information on the skeleton B. The tracking unit 24 may display the tracking result on the user terminal 30 via the display unit 27. The tracking unit 24 may also store the tracking result in the memory unit 26.

[0080] 〔program〕 A program according to one aspect of the present invention is a behavior tracking program for causing a computer to function as the behavior tracking system described above, and is a program for causing a computer to function as the acquisition unit, the division unit, the detection unit, and the tracking unit.

[0081] The control blocks (acquisition unit 21, division unit 22, detection unit 23, and tracking unit 24) of the behavior tracking device 20 provided in the behavior tracking system 100 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0082] In the latter case, the behavior tracking device 20 includes a computer that executes instructions of a program, which is software that realizes each function. The computer includes, for example, one or more processors, and a computer-readable recording medium that stores the program. The object of the present invention is achieved by the processor reading the program from the recording medium and executing it in the computer. The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transient tangible medium," such as a ROM (Read Only Memory), as well as a tape, a disk, a card, a semiconductor memory, and a programmable logic circuit. The device may further include a RAM (Random Access Memory) that expands the program. The program may be supplied to the computer via any transmission medium (such as a communication network and a broadcast wave) that can transmit the program. Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0083] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Industrial Applicability]

[0084] The present invention can be used in the field of livestock farming. [Explanation of symbols]

[0085] 10. Image sensor 20 Behavioral Tracking Device 21 Acquisition Department 22 Division 23 Detection unit 24 Tracking Department 25 Identification section 26 Memory section 27 Display section 28 Analysis Department 30 User terminal 200 Animal specimens 300 Controlled Area 400 consecutive images 500 individual images F frame B. Skeleton

Claims

1. an acquisition unit that acquires continuous images captured by an image sensor; a division unit for dividing the continuous image into individual images along a time axis; a detection unit that detects a frame surrounding an individual animal and a skeleton of the individual animal in the individual image; a tracking unit that tracks behavior of the individual animal among the plurality of individual images based on the time axis information, the frame information, and the skeletal information of the individual images; Equipped with A behavior tracking system in which the tracking unit tracks the behavior of individual animals by estimating the frame and skeleton corresponding to the same individual animal based on the similarity of the time axis information, the frame information, and the skeletal information between multiple individual images.

2. The behavior tracking system according to claim 1 , wherein in the tracking section, at least one of the individual images includes a plurality of individual animals.

3. A behavior tracking system as described in claim 1 or 2, wherein in the successive images, the individual animal does not move across a boundary of the angle of view of the imaging element.

4. In the tracking unit, the information about the frame includes information about coordinates of the frame in the individual image; 4. The behavior tracking system according to claim 1, wherein the information on the skeleton includes information on coordinates of the skeleton in the individual images and information on coordinates of the skeleton in the frame.

5. The detection unit is a frame detection unit that detects the frame surrounding the individual animal in the individual image; a skeleton detection unit that detects a skeleton of an individual animal within the frame detected by the frame detection unit; The behavior tracking system according to any one of claims 1 to 4, comprising:

6. 6. The behavior tracking system according to claim 1, wherein the frame has a rectangular shape.

7. The behavior tracking system according to any one of claims 1 to 6, further comprising an identification unit for identifying an individual animal in at least one of the individual images.

8. The behavior tracking system according to any one of claims 1 to 7, wherein the animals are livestock animals.

9. An acquisition step of acquiring continuous images captured by using an imaging element; a segmentation step for segmenting the sequence of images along a time axis into individual images; a detection step of detecting a frame surrounding an individual animal and a skeleton of the individual animal in the individual images; a tracking step of tracking a behavior of the individual animal among the plurality of individual images based on the time axis information, the frame information, and the skeleton information of the individual images; Including, A behavior tracking method in which, in the tracking step, the behavior of the individual animal is tracked by estimating the frame and skeleton corresponding to the same individual animal based on the similarity of the time axis information, the frame information, and the skeleton information between the multiple individual images.

10. A behavior tracking program for causing a computer to function as the behavior tracking system described in any one of claims 1 to 8, the program causing a computer to function as the acquisition unit, the division unit, the detection unit, and the tracking unit.

Images