Dynamic object status management system

Abstract

The present invention provides a motion state management system capable of estimating and / or predicting the state of a moving object based on images captured by a camera. [Solution] A motion state management system having a state calculation unit 12 capable of estimating and / or predicting the state of a moving object based on images captured by a motion monitoring camera and an individual identification camera, wherein the state calculation unit comprises: motion estimation means 121 that estimates the motion of the moving object based on images captured by the motion monitoring camera; tracking means 122 that tracks the moving object based on images captured by the motion monitoring camera; motion variable calculation means 123 that calculates motion variables of the moving object based on detection data from the motion estimation means and the tracking means; and learning means 124 that machine learns the motion variables of the moving object and the state of the moving object.

Description

Technical Field

[0001] The present invention relates to a moving object state management system capable of managing the state of moving objects including livestock such as humans and cows.

Background Art

[0002] Conventionally, the development of technologies for determining the health status of animals including livestock from photographed images has been underway. For example, Patent Document 1 discloses a technique for photographing an animal identified by an IC tag or the like with a camera and tracking its behavior.

[0003] Further, Patent Document 2 discloses that by analyzing video data obtained by photographing the state of cows with a camera, dryness of the nose mirror, state of excrement, abnormal drooling, abnormal nasal discharge, lameness state, and skin state are extracted, and for appetite, presence or absence of rumination, surface body temperature, presence or absence of cough, and number of steps, they are obtained as measurement data from sensors attached to individual cows.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional technologies including the above Patent Document 1 and Patent Document 2, when determining the state of livestock, information from sensors attached to the livestock in addition to the photographed image information is required, which causes a burden on the livestock due to sensor attachment and the labor involved in sensor attachment.

[0006] Note: The numbers in the patent numbers in the translation are replaced with **** as they are likely placeholders in the original text and not actual patent numbers. You may need to replace them with the correct numbers if available.In view of the above-mentioned problems, the present invention aims to provide a motion state management system that can estimate and / or predict the state of motion of motion objects, including people and livestock such as cattle, based on images captured by a camera. [Means for solving the problem]

[0007] The present invention solves the above-mentioned problems and is a motion state management system for a moving object comprising: a motion monitoring camera capable of capturing the movement of a moving object within a monitored area; an individual identification camera that captures images of the moving object in order to identify the individual moving object; and a state calculation unit capable of estimating and / or predicting the state of the moving object based on images captured by the motion monitoring camera and the individual identification camera, wherein the state calculation unit comprises: motion estimation means for estimating the movement of the moving object based on images captured by the motion monitoring camera; tracking means for tracking the moving object based on images captured by the motion monitoring camera; motion variable calculation means for calculating motion variables of the moving object based on detection data from the motion estimation means and the tracking means; and learning means for machine learning the motion variables of the moving object and the state of the moving object.

[0008] Furthermore, in the present invention, the area to be monitored is a breeding ground, and the moving object is livestock.

[0009] Furthermore, in the present invention, the motion estimation means is characterized in that the motion of the moving object is estimated by machine learning using annotations of seven classes: "standing," "sleeping," "eating," "touching," "touched," "mounting," and "mounted."

[0010] Furthermore, in the present invention, the motion variables of the moving body are characterized in that they are at least the "distance traveled", "travel time", "number of times mounted", "mounting time", "number of times mounted", "mounting time", "number of touches", "touch time", "number of times touched", "touch time", "number of times touched", "time touched", "number of times standing", "standing time", "number of times sleeping", "sleeping time", "rumination time", "number of meals", and "meal time". [Effects of the Invention]

[0011] According to the present invention, a unique effect is achieved in that the state of a moving object, including humans and livestock such as cattle, can be estimated and / or predicted solely from camera images, without the need to attach measuring devices or the like to the object. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic block diagram illustrating the system configuration in one embodiment of the present invention. [Figure 2] This is a flowchart illustrating the processing configuration for estimating and predicting the state of a moving object in one embodiment of the present invention. [Figure 3] This is an example of an image captured for motion classification in one embodiment of the present invention. [Figure 4] This is a table showing the annotation classification in one embodiment of the present invention. [Figure 5] This table shows an example of operating variables in a state estimation model according to one embodiment of the present invention. [Modes for carrying out the invention]

[0013] An embodiment of the motion state management system of the present invention will be described in detail below with reference to the attached drawings.

[0014] (System Configuration) Figure 1 shows a schematic configuration of a motion state management system according to one embodiment of the present invention, which comprises at least a motion monitoring camera 10 capable of capturing the movement of a motion object within a monitored area, an individual identification camera 11 that captures images of motion objects to identify individual motion objects, and a state calculation unit 12 capable of estimating and / or predicting the state of a motion object based on images captured by the motion monitoring camera 10 and the individual identification camera 11.

[0015] Furthermore, the state calculation unit 12 includes motion estimation means 121 that estimates the motion of a moving object based on images captured by the motion monitoring camera 10, tracking means 122 that tracks the moving object based on images captured by the motion monitoring camera 10, motion variable calculation means 123 that calculates motion variables of the moving object based on detection data from the motion estimation means 121 and the tracking means 122, and learning means 124 that machine-learns the motion variables and state of the moving object.

[0016] For example, to explain more specifically using livestock in a farm as an example of moving objects within the monitored area, one or more motion monitoring cameras 10 capable of capturing the movements of livestock throughout the entire farm, such as a cowshed, are installed in the farm. In particular, if the farm is large, installing multiple motion monitoring cameras 10 within the farm ensures that the movements of livestock are reliably captured throughout the entire farm.

[0017] In addition, one or more individual identification cameras 11 are installed to photograph the faces of livestock and identify individual animals. To ensure that the faces of livestock are reliably photographed, it is preferable to install them, for example, at watering holes. Considering that the above-mentioned motion monitoring camera 10 and individual identification camera 11 will be recording 24 hours a day, it is preferable that they be configured to record in infrared light at night.

[0018] The video data captured by the above-mentioned action monitoring camera 10 and individual identification camera 11 is output to an edge computer that functions as a state calculation unit 12 installed in the breeding farm. Then, in the edge computer, from the video data of the action monitoring camera 10, the position of the livestock and the parts of the livestock (such as the face, buttocks, etc.) are analyzed, and the actions of the livestock are estimated by the action estimation means 121. In addition, from the video data of the individual identification camera 11, the individuals of the livestock are identified from the shape of the livestock and the positions of the eyes, nose, etc. As a result, it becomes possible to estimate which livestock is performing what kind of action.

[0019] Also, in the tracking means 122, tracking of the livestock is performed based on the video data captured by the action monitoring camera 10, and the movement trajectory of the livestock can be recorded. Then, based on the detection data of the above-mentioned action estimation means 121 and tracking means 122, the action variable calculation means 123 calculates the action variables of the livestock, which will be described later.

[0020] Then, in the learning means 124, the action variables and states of the above-mentioned livestock are machine-learned, and it becomes possible to estimate and / or predict the state of the livestock based on the captured video data.

[0021] The state of the livestock estimated and / or predicted by the above-mentioned edge computer, the video data captured by the action monitoring camera 10 and the individual identification camera 11, etc. are uploaded to the cloud server 13, and in the cloud server 13, the data of the livestock are managed for each breeding farm and further for each camera that captured the video.

[0022] Also, the estimated and / or predicted state of each individual livestock is notified to the information display terminal 14 possessed by the staff of the breeding farm, and it can be confirmed at any time. Note that the information display terminal can be a PC, a smartphone, a tablet terminal, etc.

[0023] By checking the notifications on the information display terminal 14, the staff at the livestock farm can immediately observe the condition of the livestock in question. For example, if the livestock is in estrus, they can take steps to induce conception, and if a disease is suspected, they can provide treatment early.

[0024] Next, based on the flowchart shown in Figure 2, the processing flow will be explained below using livestock in the farm of this embodiment as an example.

[0025] (Livestock detection, behavior estimation, and classification) As mentioned above, the position of the livestock and parts of the livestock (face, hindquarters, etc.) are analyzed from the video data of the motion monitoring camera 10, and the motion estimation means 121 estimates the livestock's movements. In this embodiment, annotations are made for seven classes as shown in Figure 4: "standing," "sleeping," "eating," "touching," "touched," "mounting," and "mounted." Training data is created, and the learning means 124 learns the movements of the livestock. Note that "touching" and "touched" refer to the actions of one livestock resting its face on the hindquarters of another livestock or having its face resting on the hindquarters of another livestock.

[0026] As a result, as shown in Figure 3, for example, the movements of livestock are estimated and classified. In this embodiment, the movements of livestock are estimated and classified from five still images per second. By using still images, the processing load can be reduced, and movement classification can be performed quickly.

[0027] (Individual identification) As mentioned above, individual livestock identification is performed by capturing the shape of the livestock and facial features such as horns, eyes, and noses from video data of individual identification cameras 11 installed at watering holes and other locations. However, this is not necessarily the only method; individual identification can also be performed by attaching a two-dimensional code to an appropriate part of the livestock's body. The information of livestock identified in this way is carried over to tracking, and the movements of each individual livestock are tracked.

[0028] (tracking) As mentioned above, the tracking means 122 tracks livestock based on video data captured by the motion monitoring camera 10, and records the movement trajectory of the livestock. Specifically, it detects the movement trajectory of the livestock along with time information.

[0029] This makes it possible to calculate features such as the distance traveled per unit time by individual livestock. It is known that livestock tend to move more when they are in estrus, so this information is extremely useful.

[0030] Furthermore, the tracking method using the motion monitoring camera 10 can be implemented, for example, by the prior art disclosed in Japanese Patent Publication No. 7174389 by the applicant. This makes it possible to track livestock without attaching GPS devices, angular velocity / accelerometers, etc. to their bodies.

[0031] Furthermore, while livestock move by walking on four legs, in some situations the motion monitoring camera 10 may only be able to capture images of the front legs or hind legs. Therefore, in this embodiment, the distance traveled is calculated by identifying the front legs and hind legs separately.

[0032] In addition, in this embodiment, it is also possible to determine rumination based on video data captured by the motion monitoring camera 10. Specifically, rumination is detected by the amount of movement (difference between frames) in the head region of the livestock. This makes it possible to calculate the rumination time.

[0033] (State estimation, state prediction) In this embodiment, as described above, the motion variable calculation means 123 calculates the motion variables of the livestock based on the detection data of the motion estimation means 121 and the tracking means 122. Specifically, it is possible to calculate 18 motion variables from X1 to X18 shown in Figure 5, which are the livestock's "distance traveled," "travel time," "number of times mounted," "mounting time," "number of times mounted," "time spent mounted," "number of times touched," "touch time," "number of times touched," "time spent touched," "number of times standing," "standing time," "number of times sleeping," "sleeping time," "rumination time," "number of meals," "meal time," and "other time (time that could not be determined)."

[0034] Furthermore, Figure 5 shows the degree of correlation between these 18 operating variables and "estrus state," "illness," and "insufficient food intake" in four stages, from double circles to crosses. These 18 operating variables are highly effective feature quantities identified in the development of this invention.

[0035] In this embodiment, by using machine learning on the livestock's state along with the above operating variables using the learning means 124, a livestock state estimation model is created that can estimate and even predict at least four livestock states: "estrus," "poor health," "unable to stand," and "normal."

[0036] The livestock status estimated and / or predicted by the livestock status estimation model described above is notified to the information display terminal 14 held by the farm staff, as mentioned above, allowing them to immediately check the livestock status. In addition to displaying the livestock status, the information display terminal can also display past livestock footage and livestock movement trajectories.

[0037] The above describes one example of an embodiment of the motion state management system of the present invention. However, the embodiments of the present invention are not necessarily limited to the forms described above, and various modifications are possible.

[0038] For example, in the embodiment described above, as shown in Figure 1, edge computers and cloud servers were used to reduce processing load and speed up information processing, but this is not necessarily the only way to achieve this. Naturally, it is also possible to implement this using PCs and servers on a local network, and it is even possible to use application servers.

[0039] Furthermore, data can be transmitted and received between devices via wired or wireless connections, and can be done through public communication lines as well as various other known methods.

[0040] The present invention's motion state management system is particularly effective for managing the state of livestock such as cattle and pigs, but it can also be used for managing the state of other livestock and even people. For example, if the monitoring area is a school classroom, it becomes possible to estimate students' actions (such as raising their hands or falling asleep) and their state, thereby understanding their attitude towards class. In addition, when a teacher is absent, the tracking means can be effectively used to predict abnormal behavior or incidents in the classroom. Furthermore, the monitoring area is not limited to classrooms or livestock farms, but can be applied to managing the state of people in a wide range of settings, such as factory facilities and airport / port facilities.

[0041] Furthermore, the learning means 124 in the present invention can appropriately utilize a multilayer neural network created using deep learning techniques known as machine learning devices.

[0042] While embodiments of the present invention have been described above, these embodiments are provided to facilitate understanding of the present invention and do not limit it. The present invention can be modified and improved without departing from its spirit, and its equivalents are included. Furthermore, the combinations or omissions of the components described in the claims and specification are possible to the extent that at least some of the above-mentioned problems can be solved or at least some of the effects can be achieved. [Explanation of Symbols]

[0043] 10. Cameras for monitoring motion 11. Individual identification cameras 12. State calculation unit (edge ​​computer) 121 Motion estimation means 122 Tracking means 123 Means for calculating operating variables 124 Learning Methods

Claims

1. A motion monitoring camera capable of capturing the movement of moving objects within the monitored area, A camera for individual identification that takes pictures of moving objects in order to identify the individual object, A motion state management system comprising: a state calculation unit capable of estimating and / or predicting the state of the motion based on images captured by the motion monitoring camera and the individual identification camera, The state calculation unit, Motion estimation means for estimating the motion of the moving object based on images captured by the motion monitoring camera, A tracking means that tracks the moving object based on the image captured by the motion monitoring camera, A motion variable calculation means that calculates the motion variables of the moving body based on the detection data of the motion estimation means and the tracking means, The system includes the motion variables of the moving body and a learning means for machine learning the state of the moving body. A dynamic body status management system characterized by the following:

2. The area under surveillance is a breeding ground, and the moving object is livestock. The dynamic body status management system according to feature 1.

3. The motion estimation means estimates the motion of the moving object by machine learning using annotations in seven classes: "standing," "sleeping," "eating," "touching," "touched," "mounting," and "mounted." The dynamic body status management system according to feature 2.

4. The motion variables of the aforementioned moving object are at least the following: "distance traveled", "travel time", "number of times mounted", "mounting time", "number of times mounted", "mounting time", "number of times touched", "touch time", "number of times touched", "touch time", "number of times touched", "time touched", "number of times standing", "standing time", "number of times sleeping", "sleeping time", "rumination time", "number of meals", and "meal time". The dynamic body status management system according to claim 2 or 3.

Images