Image-based animal death detection device and method

Abstract

An image-based animal death detection device according to an embodiment of the present invention comprises: a communication module for receiving an image from at least one camera assigned to each livestock room; a memory in which a program for detecting animal death on the basis of an image is stored; and a processor for executing the program, wherein the program: inputs a received image into an animal detection model to output center point coordinates and posture information of each animal individual; calculates a moving distance of each animal individual on the basis of the center point coordinates; calculates a posture ratio of each animal individual every preconfigured unit time on the basis of the posture information; and inputs activity feature information, which includes the moving distance and the posture ratio of each animal individual, into a death detection model, and if an animal individual is determined to be an abnormal individual for a certain time period, detects the animal individual as dead.

Description

Image-based animal death detection device and method

[0001] The present invention relates to an image-based animal death detection device and method.

[0002] Most livestock farms rely heavily on experience and manual labor for feeding management. Farmers and workers with extensive experience in the livestock industry personally monitor the health, growth stage, and feed intake of their animals and develop feeding plans. This management approach relies heavily on skilled labor, making it difficult to base management on objective data.

[0003] Livestock farms face the dual challenges of a labor shortage and an aging workforce. Young workers are reluctant to enter the agricultural sector, making livestock management increasingly difficult. This situation, coupled with the aging workforce, risks declining productivity and deteriorating management.

[0004] The fact that feeding management on livestock farms relies primarily on experience and human resources poses significant challenges in disease control. Traditional management methods make it difficult to conduct the precise monitoring and data analysis necessary for early detection and prevention of livestock diseases. In particular, diseases that occur among livestock, if not identified in their early stages, can spread rapidly and cause serious damage to the entire farm.

[0005] If a disease outbreak is not promptly addressed, it can lead to significant economic losses beyond simply animal health issues. This can lead to increased treatment costs, decreased productivity, and, in the worst-case scenario, even mass livestock mortality. This can severely impact farm management and sustainability.

[0006] In this regard, prior art literature includes Korean Patent Publication No. 10-2023-0046156 (Title of invention: Artificial intelligence-based livestock abnormal behavior pattern extraction method and device).

[0007] The present invention is intended to solve the above-mentioned problem, and a technical task is to provide an image-based animal death detection device and method that clusters animal individuals into groups with similar activity patterns by considering the movement distance and posture ratio of the animal individuals per unit time, and determines an animal individual that falls below the standard value of each group as an abnormal individual.

[0008] However, the technical task that this embodiment seeks to achieve is not limited to the technical task described above, and other technical tasks may exist.

[0009] As a technical means for solving the above-described technical problem, an image-based animal death detection device according to a first aspect of the present invention comprises: a communication module for receiving an image from at least one camera assigned to each livestock barn; a memory storing a program for detecting animal death based on an image; and a processor for executing the program, wherein the program inputs the received image into an animal detection model to output center point coordinates and posture information of each animal, calculates a moving distance of each animal based on the center point coordinates, calculates a posture ratio of each animal based on the posture information at a preset unit time, and inputs activity feature information including the moving distance and posture ratio of each animal to the death detection model, and detects the animal as dead when the animal is determined to be an abnormal animal for a certain period of time.

[0010] In addition, a method for detecting animal death based on an image using an animal death detection device according to a second aspect of the present invention includes the steps of: inputting an image received from at least one camera assigned to each livestock pen into an animal detection model and outputting center point coordinates and posture information of each animal; calculating a moving distance of each animal based on the center point coordinates; calculating a posture ratio of each animal at a preset unit time based on the posture information; and inputting activity feature information including the moving distance and posture ratio of each animal into the death detection model and detecting the animal as dead if the animal is determined to be an abnormal animal for a predetermined period of time.

[0011] According to the aforementioned means of solving the problem of this invention, it is possible to more accurately understand the normal behavioral patterns of animals through analysis reflecting circadian rhythms, detect abnormal behavior more reliably, thereby increasing the accuracy of animal death detection and improving the efficiency of livestock management.

[0012] In particular, by setting activity pattern criteria that take into account the circadian rhythm of each time zone, abnormal individuals that fall below or exceed the criteria for a certain period of time can be identified, and animal individuals that are consistently identified as abnormal for a certain period of time can be detected as dead.

[0013] FIG. 1 is a block diagram illustrating the configuration of an image-based animal death detection device according to one embodiment of the present invention.

[0014] FIG. 2 is a flowchart illustrating an image-based animal death detection method according to one embodiment of the present invention.

[0015] FIG. 3 is a drawing for explaining the operation of an image-based animal death detection device according to one embodiment of the present invention.

[0016] FIG. 4 is a drawing for explaining detailed information according to one embodiment of the present invention.

[0017] Figures 5 and 6 are drawings for explaining an activity pattern according to one embodiment of the present invention.

[0018] FIG. 7 is a graph illustrating an activity pattern for a pig's day-cycle according to one embodiment of the present invention.

[0019] FIG. 8 is a diagram showing a process of updating the reference value of each activity pattern of a dead body detection model according to one embodiment of the present invention.

[0020] FIGS. 9 to 11 are drawings for explaining an animal detection model of an image-based animal death detection device according to one embodiment of the present invention.

[0021] Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in the present specification, and the technical ideas disclosed in the present specification are not limited by the attached drawings. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and the size, shape, and shape of each component shown in the drawings can be variously modified. The same / similar drawing reference numerals are assigned to the same / similar parts throughout the specification.

[0022] The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if a detailed description of a related known technology is judged to obscure the gist of the embodiments disclosed in this specification, the detailed description thereof has been omitted.

[0023] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0024] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component."

[0025] FIG. 1 is a block diagram illustrating a configuration of an image-based animal death detection device according to one embodiment of the present invention, and FIG. 2 is a flowchart for explaining an image-based animal death detection method according to one embodiment of the present invention.

[0026] Referring to FIG. 1, the image-based animal death detection device (100) includes a communication module (110), a memory (120), and a processor (130), and may further include a database (140). The image-based animal death detection device (100) receives real-time captured images through multiple CCTVs or other various cameras placed in a livestock barn, and performs an operation of detecting the death of an individual animal using the images.

[0027] To this end, the image-based animal death detection device (100) can be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, laptop, etc. equipped with a web browser, and the portable terminal can include, for example, all kinds of handheld-based wireless communication devices such as various smart phones, tablet PCs, smart watches, etc., which are wireless communication devices that ensure portability and mobility. In addition, the image-based animal death detection device (100) can function as a server that provides the result of animal death detection for an animal subject image received from the outside. When the image-based animal death detection device (100) operates as a server, the server can operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). Additionally, servers can be deployed in the form of private clouds, public clouds, or hybrid clouds.

[0028] A network is a connection structure that enables information exchange between each node, such as terminals and devices, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), LiFi, etc.

[0029] The communication module (110) receives images of a subject from at least one camera (10) assigned to each livestock pen. The subject includes various types of animals such as cows, pigs, dogs, etc. The communication module (110) may include a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired or wireless connections with other network devices.

[0030] The memory (120) stores a program for detecting image-based animal death from an input image taken of a target object. At this time, the program for detecting image-based animal death inputs the received image into an animal detection model to output center point coordinates and posture information of each animal, calculates the movement distance of each animal based on the center point coordinates, calculates the posture ratio of each animal at a preset unit time based on the posture information, and inputs activity feature information including the movement distance and posture ratio of each animal into the death detection model to detect the animal as dead if it is determined to be an abnormal animal for a certain period of time.

[0031] At this time, the input image may be an image received through the communication module (110). The animal detection model is learned based on learning data that matches multiple images containing one or more animal objects and object detection information for the animal objects included in each image. In addition, the object detection information is information on a bounding box formed to fit the animal object detected in the image, and includes the center point coordinates of the bounding box, the width of the bounding box, the length of the bounding box, feature point information indicating one end of the animal object, and pose information. The specific contents of the animal detection model and the object detection information will be described later.

[0032] At this time, the memory (120) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain the stored information. The memory (120) may perform a function of temporarily or permanently storing data processed by the processor (130). In addition to a volatile storage device that requires power to maintain the stored information, the memory (120) may include a magnetic storage media or a flash storage media, but the scope of the present invention is not limited thereto.

[0033] The processor (130) executes a program for detecting animal death based on images stored in the memory (120), and outputs a death detection result for the target object as a result of the execution.

[0034] In one example, the processor (130) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0035] The database (140) can store various data for learning an animal detection model or images captured by a camera (10) received through a communication module (110). In particular, images captured by each camera (10) installed at each location in each livestock shed can be stored separately. In addition, the database (140) cumulatively stores object detection information extracted by a program (hereinafter, “program”) for detecting image-based animal death, and based on this object detection information, it can be utilized in various applications for monitoring the condition of animals.

[0036] FIG. 3 is a drawing for explaining the operation of an image-based animal death detection device according to one embodiment of the present invention.

[0037] Referring to FIGS. 2 and 3, the program inputs the received image into the animal detection model (200) and outputs object detection information including the center point coordinates and posture information of each animal object (S110).

[0038] At this time, a detailed description of the animal detection model (200) that outputs object detection information including the center point coordinates and posture information of each animal object from the input image will be described later with reference to FIGS. 9 to 11.

[0039] For example, the first through nth cameras can be installed at different locations in each livestock shed and monitor the animals in each shed. Furthermore, the footage captured by each camera can be collected separately.

[0040] Next, the program tracks each animal individual and calculates the movement distance of each animal individual based on the coordinates of the center point of each animal individual (S120).

[0041] The program assigns an identification number to each animal detected through the animal detection model (200), tracks each animal using the degree to which the center coordinates of each identified animal are adjacent to each other, and calculates the movement distance of each animal by accumulating and storing the center coordinates of each animal for each identification number as they move for each frame within the input image.

[0042] For example, the program can use a conventional object tracking algorithm to assign an identification number (ID) to each animal and track each animal by using the degree to which the center coordinates of each identified animal are adjacent to each other. In addition, the center coordinates of each animal by identification number (ID) are accumulated and stored as they move for each frame in the input image, and the movement distance of each animal can be calculated using mathematical equation 1.

[0043] <Mathematical Formula 1>

[0044]

[0045] Here, t is the frame unit, p2c is the scale for converting pixel unit to cm unit, and 1h is the conversion of 1 hour to frame unit, such as 60*60*30=108,000f when the frame rate of the camera is 30fps.

[0046] Here, the movement distance is calculated by tracking the coordinates of the center point of the animal object in each frame, calculating the movement distance between each frame, and then accumulating and adding these values ​​in chronological order, and means the total movement distance moved by each animal object over a certain period of time.

[0047] For example, conventional object tracking algorithms include SORT (simple online and real-time tracking algorithm) based on the Hungarian algorithm, Deep SORT, ByteTrack (high-performance multi-object tracking), and correlation-based tracking. Furthermore, the program can track the degree of overlap between the regions of detected animal objects for each frame using a Kalman filter to determine whether the detection results between two frames are of the same object.

[0048] FIG. 4 is a drawing for explaining detailed information according to one embodiment of the present invention.

[0049] Next, the program calculates the posture ratio of each animal individual at a preset unit time based on the posture information (S130).

[0050] The program can calculate the posture ratio of each animal individual by dividing the sum of the number of frames in which the lying posture is detected during a unit time among the posture information of each animal individual by the total number of frames in which posture information is detected during a unit time.

[0051] For example, posture information can be divided into standing and lying postures as shown in Fig. 4(a). In particular, the lying posture can be divided into three postures, including sitting as shown in Fig. 4(b), lateral lying as shown in Fig. 4(c), and sternal lying as shown in Fig. 4(d).

[0052] For example, assuming a frame rate of 30 fps for a unit time of 1 minute, the program can calculate the posture ratio by dividing the sum of the number of frames of the lying posture (steral + lateral + sitting) of each animal individual (ID) detected through the animal detection model (200) by the total number of frames (60 s × 30 fps) of posture information (steral + lateral + sitting + standing) for a unit time.

[0053] That is, if the lying posture ratio is greater than or equal to a threshold (e.g., 50%) of the total number of frames, it means that the posture ratio is high, and if the lying posture ratio is less than or equal to the threshold of the total number of frames, it means that the posture ratio is low. For example, as illustrated in FIG. 6 described below, if the sum of the frames of the lying posture (steral + lateral + sitting) is 90% of the total number of frames, the posture ratio is greater than or equal to the threshold, which is indicated as a high posture ratio, and if the sum of the frames of the lying posture is 10% of the total number of frames, the posture ratio is less than the threshold, which is indicated as a low posture ratio.

[0054] For example, the program can manage data (activity characteristic information), including the movement distance and posture ratio of each animal (ID) generated in steps S120 and S130, for each livestock room where each camera is installed and by time zone. For example, the program can analyze images captured in a specific livestock room at a specific time zone and record the movement distance and posture ratio of each animal during that time.

[0055] Finally, the program inputs activity characteristic information including the movement distance and posture ratio of each animal into the death detection model (300) and detects the animal as dead if it is determined to be an abnormal animal for a certain period of time (S140).

[0056] For example, the dead animal detection model (300) is a clustering model constructed by clustering animal individuals into activity patterns based on the movement distance and posture ratio of the animal individuals at each time, and classifies the activity characteristic information (movement distance, posture ratio) into activity patterns, and can determine an animal individual that falls below the standard value of each activity pattern at each time as an abnormal individual.

[0057] For example, a method for constructing a death detection model (300) may include a data collection step, a data calculation step, and a clustering step. First, the data collection step may collect frame-by-frame data (posture details, center point coordinates) every hour through an animal detection model. Next, the data calculation step may calculate the lying posture ratio (number of lying posture frames / number of observed frames) and the movement distance (Mathematical Formula 1) based on the collected data as described above. Accordingly, two clustering variables may be obtained. Next, the clustering step may construct a death detection model (300) that classifies animals into three groups by repeating the initial setup, assignment, update, and convergence processes. For example, in the initial setup step, K-means clustering may be used to divide animals into three clusters, and the initial cluster centers calculated in the experiment may be used in the first operation. Next, in the assignment step, each data point may be assigned to the nearest cluster center, and the Euclidean distance may be used to calculate the distance between each data point and the cluster center. Next, in the update process, the center of each cluster is recalculated, and the average of all data points assigned to the cluster can be calculated and set as the new cluster center. Subsequently, in the convergence process, the assignment and update processes can be repeated until the cluster center no longer changes. Consequently, each data point belongs to one of the three clusters. Therefore, the death detection model (300) can classify the two variables (lying posture ratio / movement distance) into three groups. The three groups include a low-activity group, a feeding group, and a high-activity group, which correspond to the activity patterns described below.

[0058] The activity patterns described below can be classified according to commonly understood criteria.

[0059] Figures 5 and 6 are drawings for explaining an activity pattern according to one embodiment of the present invention.

[0060] Referring to FIG. 5, the activity pattern may include a low-activity pattern (401), a high-activity pattern (402), and a feeding pattern (403). For example, the low-activity pattern (401) is a pattern that appears during a sleep period, and refers to a state in which an animal lies down and sleeps with little movement. The high-activity pattern (402) is a pattern that appears during an activity period, and refers to a state in which an animal moves a lot and stands. The feeding pattern (403) is a pattern that appears during a feeding period, and refers to a state in which an animal moves little and stands around a feeder.

[0061] Specifically, as illustrated in FIG. 6, a low-energy pattern (401) may be a cluster of activity characteristic information in which the lying posture ratio is greater than or equal to a threshold and the moving distance is less than or equal to a threshold, a high-energy pattern (402) may be a cluster of activity characteristic information in which the lying posture ratio is less than or equal to a threshold and the moving distance is greater than or equal to a threshold, and a feeding pattern (403) may be a cluster of activity characteristic information in which the lying posture ratio is less than or equal to a threshold and the moving distance is less than or equal to a threshold.

[0062] FIG. 7 is a graph illustrating an activity pattern for a pig's day-cycle according to one embodiment of the present invention.

[0063] For example, for a circadian image of a pig colony, when activity characteristic information divided into one-hour units is input into a death detection model (300), the death detection model (300) can classify the activity characteristic information into each activity pattern and determine an animal individual that falls below the standard value of each activity pattern at each hour as an abnormal individual.

[0064] For example, as illustrated in Figure 7, during the sleep timeframe (t+10), it is normal for a typical animal to exhibit a low-energy pattern. Therefore, while it is normal for many animals to exhibit a low-energy pattern during the sleep timeframe, if an individual exhibits a high-energy pattern, that individual may be judged as an abnormality.

[0065] As another example, the animal death detection device (100) according to the present invention can receive input from the user regarding a key observation point. For example, if the key observation point is set to an activity time zone (t+2) with a large number of high-activity pattern individuals, the program can monitor for the occurrence of abnormal low-activity pattern individuals during the activity time zone and provide a notification to the user.

[0066] FIG. 8 is a diagram showing a process of updating the reference value of each activity pattern of a dead body detection model according to one embodiment of the present invention.

[0067] For example, the mortality detection model (300) can classify the input activity characteristic information into a low-activity pattern, a high-activity pattern, and a feeding pattern. For example, as illustrated in FIG. 8(a), the mortality detection model (300) that monitors a pigsty or livestock pen can classify the activity characteristic information into each activity pattern using an initial reference value set by experimental data. Thereafter, as illustrated in FIGS. 8(b) and 8(c), the reference value of the low-activity pattern can be updated to adapt to the pigsty or livestock pen over a certain period of time. In this way, the mortality detection model (300) according to the present invention can accurately determine abnormalities in individual animals classified as low-activity patterns by adapting to pigsty and livestock pens (cells) with different environments. Similarly, the reference values ​​of the high vitality pattern at 30 days of age as shown in Fig. 8(d), the reference values ​​of the high vitality pattern at 60 days of age as shown in Fig. 8(e), and the reference values ​​of the high vitality pattern at 90 days of age as shown in Fig. 8(f) can be updated. That is, it can be seen that the total movement distance of the reference values ​​of the high vitality pattern decreases as the pig ages. In this way, the death detection model (300) according to the present invention can accurately determine abnormalities in animal individuals classified as high vitality patterns by adapting to the vitality that inevitably decreases as the pig grows.

[0068] Below, we will examine an animal detection model (200) that generates object detection information.

[0069] FIGS. 9 to 11 are drawings for explaining an animal detection model of an image-based animal death detection device according to one embodiment of the present invention.

[0070] The program inputs an input image into the animal detection model (200) and extracts object detection information including center point coordinates and posture information.

[0071] The animal detection model (200) used in the present invention is constructed based on learning data that matches a plurality of images containing at least one animal object and object detection information1 (information on bounding boxes, feature point information, and pose information) included in each image. After being trained through a learning process, the animal detection model (200) can automatically output object detection information for an input image through an inference process for an actual input image.

[0072] The learning data used in the learning process of the animal detection model (200) includes multiple images and object detection information matched to each image. At this time, the object detection information is manually extracted for each image. That is, an expert can manually review each image and input object detection information using an appropriate SW tool, or an expert can automatically input the information using an existing animal detector that has been fully developed and then modify / supplement it. For example, the expert displays a bounding box for each animal included in the image, considering the direction of rotation relative to the reference axis of the animal, and generates information such as the coordinates of the center point of each bounding box, the width of the bounding box, the length of the bounding box, and the angle at which the bounding box is rotated with respect to the reference axis. In addition, the worker can additionally extract information on the type and posture of the animal and utilize it as learning data.

[0073] At this time, the object detection information includes information about a bounding box formed to fit an animal object detected in an input image, and information about the feature points, type, and posture of the animal object.

[0074] For example, the object detection information may include information about a bounding box (rbbox) formed to fit an animal object, such as the center point coordinates (xc, yc) of the bounding box, the width (w) of the bounding box, the length (h) of the bounding box, and the angle information (theta) by which the bounding box is rotated with respect to the reference axis. In addition, the object detection information may include position information about keypoints of the animal object, such as the position of the end of the head of the animal object (x1, y1), the position of the neck (x2, y2), and the position of the end of the torso (xn, yn).

[0075] Furthermore, object detection information refers to the type (class) of an animal, distinguishing it into different species such as cows, pigs, and dogs, but is not limited thereto. For example, even within the same species, types can be distinguished by growth stage. In the case of pigs, types can be distinguished such as suckling piglets, weaned piglets, growing pigs, fattening pigs, gilts, pregnant pigs, and farrowing pigs.

[0076] Additionally, object detection information is information about the pose of an animal object, which can be categorized into standing, sitting, lateral lying, and lying forward, including sternal lying.

[0077] Referring to FIG. 9, the animal detection model (200) may include a back part (210), a neck part (220), and a head part (230).

[0078] The backbone (210) extracts features from input images and is a commonly used component in deep neural network-based image analysis and processing methods. As illustrated, the backbone (210) primarily takes the form of a 2D convolutional accumulation, and has been enhanced to incorporate various neural network structures to enhance its efficiency. Backbones of various structures commonly receive images and extract intermediate information, which is then transmitted to the neck (220).

[0079] The neck unit (220) collects intermediate information from each layer of the back unit (210) based on the features extracted from the back unit (210). The neck unit (220) is a lower neural network that constitutes a universal object detector and performs the role of collecting and interpreting intermediate information for each layer of the back unit (210). Since the resolution of the image interpreted for each layer is different, the neck unit (220) extracts intermediate information for each layer to effectively detect animals of various sizes depending on the body type of the animal, whether the target is far away or close, and provides the extracted intermediate information to the head unit (230). The specific configuration of the neck unit (220) varies depending on the shape of the back unit (210) described above, and the number of layers and hyper parameters for each layer of the specific neural network constituting the neck unit (220) may vary depending on the shape of the back unit (210).

[0080] The head unit (230) outputs object detection information based on the intermediate information collected from the neck unit (220). The head unit (230) receives the intermediate information obtained from the neck unit (220) and outputs object detection information. The head unit (230) receives the intermediate information of each layer of the neck unit (220) and outputs object detection information recognized for each layer. In particular, the head unit (230) of the present invention includes a plurality of animal detection subnets, and each animal detection subnet includes a subnet for extracting a bounding box and feature points, a subnet for extracting an animal type, and a subnet for extracting an animal posture, as shown in FIG. 10.

[0081] Meanwhile, an NMS (Non-maximum Suppression) module may be further combined at the output terminal of the head unit (230). This is an algorithm for selecting a bounding box with the highest similarity when multiple bounding boxes are created for the same object, and since it corresponds to prior art, a detailed description thereof will be omitted.

[0082] The subnet that extracts bounding boxes and feature points consists of a "cascaded multi-lane deep convolutional network." The cascaded multi-lane deep convolutional network is constructed according to a causal order to find bounding boxes and feature points for a given animal image. To define a single object detection information in each image, the following causal order is followed.

[0083] That is, as illustrated in Figure 11, first, the center point (Xc, Yc) and feature point information (nose, neck, hip) are indicated. Next, a tangent line crossing the center point and one or more points is drawn. Finally, the area (surface) through which the tangent line passes through the center is specified.

[0084] In this structure, the cascaded multipath deep convolutional network transmits information according to the causal order mentioned above and outputs each piece of information. That is, the first path outputs the center point and key points, the second path outputs the direction of the tangent (theta), and the third path outputs the width (w) and height (h) of the region (bounding box) containing the tangent and center point (xc, yc).

[0085] Meanwhile, the subnet that extracts animal types and the subnet that extracts animal poses are each obtained through a general structure, that is, a single-pass deep convolutional network. In addition, the pose extraction algorithm can be a 1-stage method that utilizes a multi-object detection algorithm such as YOLO or DETR that directly learns the type of pose from images for each pose. In addition, the pose extraction algorithm can be a 2-stage method that first extracts keypoints for each object using technologies such as OpenPose, DeepLabCut, and YoLoPose that estimate keypoints for each object, and then utilizes a DNN or RNN model that distinguishes poses based on the composition and changes of these keypoints. In addition, the pose extraction algorithm can be a 2-stage method that detects the region of each object using an object detection algorithm and then classifies the image within the detected region using an image classification algorithm such as ResNet or EfficientNet to classify the pose.

[0086] An animal detection model (200) like this can also be expressed in the following mathematical formula.

[0087]

[0088] Here, A={RK,c,p} denotes vectorized object detection information, M(x) denotes an animal detection model, I denotes an input image matrix (having dimensions of image width x image height x image channel), and E(A) denotes encoded animal detection information. In addition, B(x), N(x), and H(x) denote the back part (210), the neck part (220), and the head part (230), respectively.

[0089] When the input image matrix is ​​input to the animal detection model M(x), the output is trained to be identical to the encoded animal detection information E(A), and the animal detection model is built through a process in which the weights of the animal detection model are repeatedly updated through backpropagation learning.

[0090] The training data used in the animal detection model training process includes multiple images and object detection information matched to each image. Some of the object detection information may be manually extracted for each image. For example, a worker may review each image, use an appropriate software tool to identify a bounding box, and manually input information such as the center point coordinates, width, and length of the bounding box. Alternatively, a previously developed animal detector may be used to automatically input the information, which may then be modified or supplemented by a human. For example, the worker may mark a bounding box for each animal in the image, considering its rotation relative to the reference axis. The worker may then generate the center point coordinates, width, and length of each bounding box using a preprocessing algorithm. Then, a preprocessing algorithm may be used to generate angle discretization information, which discretizes the rotation angle of the bounding box and includes this angle discretization information in the training data. Furthermore, the worker may additionally extract information regarding the animal's type or posture for use as training data. The object detection information included in the training data undergoes an encoding process before being used in the training process. The object detection information may be encoded through the following process:

[0091] First, the area of ​​interest (230) by head part ) is specified. At this time, the area of ​​interest ( ) is defined as the processing area x size type x angle type x box ratio for each head part (230).

[0092] Additionally, the degree of overlap (o) between the animal area and the area of ​​interest recorded in the object detection information (A) a,k ) is calculated using the following mathematical formula.

[0093]

[0094] Here, IoU(x,y) calculates the degree of overlap between two bounding boxes.

[0095] Next, the animal region with the highest degree of overlap for each region of interest (R k ) only. Here, k' = argmax k (o a,k )am.

[0096] Next, encoding is performed between the region of interest and the corresponding animal region.

[0097]

[0098] At this time, is processed as follows:

[0099] When,

[0100] As such, it is processed.

[0101] Accordingly, the output is as follows.

[0102]

[0103] likewise, is processed as follows:

[0104] When,

[0105] It is processed as follows.

[0106] Accordingly, the output is as follows.

[0107]

[0108] Through the above process, encoded detection information is used in the process of building an animal detection model.

[0109] Meanwhile, backpropagation learning can be used in the learning process of the animal detection model. That is, the loss value between the encoded object detection information E(A) and its estimate is calculated, and the process of updating the neural network parameters that constitute the animal detection model to reduce this loss value is repeated. For example, when calculating the loss value of the bounding box (rbbox) and the feature points (keypoints) of an animal object, the L1 or L2 loss can be used, and the loss value of the animal object type (c) or animal object pose (p) can be a discriminative loss such as binary cross entropy loss or focal loss.

[0110] Using this loss function, an animal detection model is built by repeating learning until the total loss falls below the target value.

[0111] In this way, we will examine the process of inferring object detection information (A) for an input image using the constructed animal detection model (M(x)). This can be expressed mathematically as follows.

[0112]

[0113] That is, when an input image is input to an animal detection model, encoded detection information (E(A)) can be obtained. Accordingly, a process of decoding the encoded detection information is performed.

[0114] In, c a Only the values ​​when c is higher than the threshold are left. In other words, a' = a if c a > thr, where thr is the detection threshold.

[0115] and, Each of these is performed to obtain decrypted animal detection area and feature point information. The processing process is as follows.

[0116] When,

[0117] It is processed as follows.

[0118] Accordingly, the output is as follows.

[0119]

[0120] likewise, is processed as follows:

[0121] , When,

[0122] It is processed as follows.

[0123] Accordingly, the output is as follows.

[0124]

[0125] The object detection information (R, K, c, p) output through the decryption process may contain multiple overlapping pieces of information for a single animal. To address this, an algorithm that removes overlapping animal detection information can be applied.

[0126] In this way, object detection information can be overlaid on the image to visually confirm animal detection information.

[0127] Below, the description of the same configuration among the above-described configurations is omitted.

[0128] Again, referring to FIG. 2, the image-based animal death detection method using the animal death detection device includes a step (S110) of inputting an image received from at least one camera assigned to each livestock pen into an animal detection model (200) to output center point coordinates and posture information of each animal, a step (S120) of calculating a moving distance of each animal based on the center point coordinates, a step (S130) of calculating a posture ratio of each animal based on the posture information at a preset unit time, and a step (S140) of inputting activity feature information including the moving distance and posture ratio of each animal into the death detection model (300) to detect the animal as dead if it is determined to be an abnormal animal for a certain period of time.

[0129] The animal detection model (200) is a model learned using learning data labeled with center point coordinates of a bounding box formed to fit an animal object included in each image, feature point information of the animal object, and posture information, and can output center point coordinates and posture information of each animal object for the image.

[0130] The step of calculating the movement distance (S120) may include a step of assigning an identification number to each animal detected through the animal detection model (200), a step of tracking each animal using the degree to which the center coordinates of each identified animal are adjacent to each other, and a step of calculating the movement distance of each animal by accumulating and storing the center coordinates of each animal for each identification number as they move for each frame in the input image.

[0131] The step of calculating the posture ratio (S130) may include a step of calculating the posture ratio of each animal by dividing the sum of the number of frames in which a lying posture is detected during a unit time among the posture information by the total number of frames in which posture information is detected during a unit time.

[0132] At this time, the posture information can be divided into standing posture, sitting posture, lateral lying posture, and sternal lying posture.

[0133] The death detection model (300) is a clustering model constructed by clustering animal individuals into activity patterns based on the movement distance and posture ratio of the animal individuals at each time, and can classify activity characteristic information into activity patterns and determine animal individuals that fall below the standard value of each activity pattern at each time as abnormal individuals.

[0134] For example, the activity pattern may include a low-energy pattern, a high-energy pattern, and a feeding pattern. Furthermore, the low-energy pattern may have a recumbent ratio above a threshold and a distance traveled below a threshold, the high-energy pattern may have a recumbent ratio below a threshold and a distance traveled above a threshold, and the feeding pattern may have a recumbent ratio below a threshold and a distance traveled below a threshold.

[0135] The animal death detection method described above can also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. The computer-readable medium can be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, the computer-readable medium can include computer storage media. The computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0136] Those skilled in the art will appreciate that, based on the above description, the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

Claims

1. In a video-based animal death detection device, A communication module for receiving images from at least one camera assigned to each livestock room; A memory storing a program for detecting image-based animal mortality; and Including a processor for executing the above program, The above program is an image-based animal death detection device that inputs a received image into an animal detection model to output center point coordinates and posture information of each animal, calculates a moving distance of each animal based on the center point coordinates, calculates a posture ratio of each animal for a preset unit time based on the posture information, and inputs activity feature information including the moving distance and posture ratio of each animal into a death detection model to detect the animal as dead if it is determined to be an abnormal animal for a certain period of time.

2. In paragraph 1, The above animal detection model is a model learned using learning data labeled with a plurality of images including at least one animal object, the center point coordinates of a bounding box formed to fit the animal object included in each image, feature point information of the animal object, and posture information, and outputs the center point coordinates and posture information of each animal object for the image.

3. In paragraph 1, The above program assigns an identification number to each animal detected through the animal detection model, tracks each animal using the degree to which the center coordinates of each identified animal are adjacent to each other, and calculates the movement distance of each animal by accumulating and storing the center coordinates of each animal for each identification number as they move for each frame within the input image, and is an image-based animal death detection device.

4. In paragraph 1, The above program calculates the posture ratio of each animal by dividing the sum of the number of frames in which the lying posture is detected during a unit time among the posture information by the total number of frames in which the posture information is detected during a unit time. An image-based animal death detection device, wherein the above posture information is classified into standing posture, sitting posture, lateral lying posture, and sternal lying posture.

5. In paragraph 1, The above-mentioned company detection model is, A clustering model constructed by clustering animal individuals into activity patterns based on the movement distance and posture ratio of the animal individuals at each time, classifying the activity characteristic information into the activity pattern, and determining an animal individual that falls below the standard value of each activity pattern at each time as an abnormal individual.

6. In paragraph 5, The above activity patterns include a low-energy pattern, a high-energy pattern, and a feeding pattern. The above low-energy pattern is one in which the lying posture ratio is above the threshold and the movement distance is below the threshold. The above high-energy pattern is one in which the lying posture ratio is below the threshold, the movement distance is above the threshold, and An image-based animal death detection device, wherein the feeding pattern is such that the lying posture ratio is less than a threshold and the movement distance is less than a threshold.

7. In a method for detecting animal death based on an image using an animal death detection device, A step of inputting images received from at least one camera assigned to each livestock room into an animal detection model and outputting center point coordinates and posture information of each animal individual; A step of calculating the movement distance of each animal based on the coordinates of the center point; A step of calculating the posture ratio of each animal individual at each preset unit time based on the above-mentioned posture information; and An image-based animal death detection method, comprising a step of inputting activity characteristic information including the movement distance and posture ratio of each animal individual into a death detection model and detecting the animal individual as dead if the animal individual is determined to be an abnormal individual for a certain period of time.

8. In paragraph 7, The animal detection model is a model learned using learning data labeled with a plurality of images including at least one animal object and the center point coordinates of a bounding box formed to fit the animal object included in each image, feature point information of the animal object, and posture information, and outputs the center point coordinates and posture information of each animal object for the image.

9. In paragraph 7, The step of calculating the above movement distance is: An image-based animal death detection method, comprising: a step of assigning an identification number to each animal detected through the animal detection model; a step of tracking each animal using the degree to which the center coordinates of each identified animal are adjacent to each other; and a step of accumulating and storing the center coordinates of each animal for each identification number as they move for each frame in the input image, thereby calculating the movement distance of each animal.

10. In paragraph 7, The step of calculating the above detailed ratio is: Including a step of calculating the posture ratio of each animal individual by dividing the sum of the number of frames in which the lying posture is detected during a unit time among the above posture information by the total number of frames in which posture information is detected during a unit time, An image-based animal death detection method, wherein the above posture information is divided into standing posture, sitting posture, lateral lying posture, and sternal lying posture.

11. In paragraph 7, The above-mentioned company detection model is, A method for detecting animal death based on an image, wherein a clustering model is constructed by clustering animal individuals into activity patterns based on the movement distance and posture ratio of the animal individuals at each time, classifying the activity characteristic information into the activity pattern, and determining an animal individual that falls below the standard value of each activity pattern at each time as an abnormal individual.

12. In paragraph 11, The above activity patterns include a low-energy pattern, a high-energy pattern, and a feeding pattern. The above low-energy pattern is one in which the lying posture ratio is above the threshold and the movement distance is below the threshold. The above high-energy pattern is one in which the lying posture ratio is below the threshold, the movement distance is above the threshold, and An image-based animal death detection method, wherein the feeding pattern has a lying posture ratio below a threshold and a moving distance below a threshold.

13. A non-transitory computer-readable recording medium having recorded thereon a computer program for performing an image-based animal death detection method according to any one of Articles 7 to 12.

Images