Apparatus for farrow determination based on behavior recognition of sow, and operational method therefor

The device employs DCNN and video recognition to accurately determine sow farrowing, addressing the challenge of missed care times and false positives, thereby improving piglet survival and farm efficiency.

WO2024242282A1PCT designated stage expired Publication Date: 2024-11-28INTELLIVIX CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/001135
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-01-24
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Current systems lack an effective method to accurately determine farrowing in sows, leading to missed golden times for piglet care and reduced survival rates due to limitations in monitoring multiple sows simultaneously and distinguishing actual farrowing events from false positives.

Method used

A device using DCNN-based technology and a video recognition module to analyze RGB images and video frames, determining the sow's standing or lying state, and employing a classification system to confirm actual farrowing events, reducing false positives through real-time monitoring with overhead cameras.

Benefits of technology

Enhances the accuracy of farrowing detection, improves piglet survival rates by enabling timely intervention and reducing worker fatigue, thereby increasing efficiency and economic profit in pig farming.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024001135_28112024_PF_FP_ABST
    Figure KR2024001135_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to an apparatus for farrow determination based on behavior recognition of a sow, and an operational method for the apparatus. The apparatus for farrow determination based on behavior recognition of a sow, according to an embodiment of the present invention, may comprise: a communication interface unit for receiving captured images of a sow preparing for farrowing; and a control unit for analyzing the received captured image so that, if it is determined that the sow is lying down, whether the detection is erroneous due to the surrounding environment or the sow is actually farrowing determined on the basis of farrowing behavior recognition of the sow.
Need to check novelty before this filing date? Find Prior Art

Description

Device for judging parturition based on behavioral recognition of sows and method for operating the device

[0001] The present invention relates to a device for judging farrowing based on behavior recognition of a sow and a method for operating the device, and more particularly, to a device for judging farrowing based on behavior recognition, which receives a real-time overhead camera image, detects the standing / lying / farming state of a sow scheduled to farrow using a DCNN (Deep Convolutional Neural Network)-based technology, and when a farrowing event of a sow determined to be in a lying state is detected, a video recognition module is used to ultimately determine whether the detected farrowing event is an actual farrowing of a piglet or a false detection event due to the surrounding environment, and a method for operating the device.

[0002] Pigs, a key livestock species in the livestock industry, are multiparous, giving birth to 8 to 12 piglets. Due to this characteristic, the health of the sow and the survival rate of the piglets born during farrowing are critical factors influencing the profitability of the pig farming industry. Therefore, monitoring the condition of the sow and providing proper care for newborn piglets during farrowing are essential to ensure piglet survival. However, workers have limited ability to constantly monitor the condition of numerous sows and to timely identify the onset of farrowing. This limitation often leads to multiple farrowings, which can lead to missed opportunities for piglet care and lower piglet survival rates.

[0003] Previously, a sow farrowing notification system was disclosed that detects the start of farrowing by obtaining and utilizing 3D color data and temperature images to predict the time of farrowing and classify piglets. In addition, a general description of an artificial intelligence processor-based sow management device, system, and method was disclosed, along with content for calculating the number of standings and standing times based on the results of a neural network processor detecting the sow's standing state. Although there is no mention of the principle of detecting the sow's standing / lying state, it does include information on a detector using DCNN. Furthermore, although farrowing is detected through image processing, there is no separate system for increasing the accuracy of the detected farrowing event. As such, prior art, including prior literature, does not have a technology for detecting the standing / lying state of a sow using DCNN-based technology and for making a final judgment and classification of the actual farrowing status through a behavior recognition model.

[0004] Accordingly, as in the embodiment of the present invention, there is a need for the development of a technology to input an image of an actual sow using RGB images, confirm the standing / lying state and whether or not actual delivery has occurred, and finally determine whether or not actual delivery has occurred through a classification system using a behavior recognition model.

[0005] The purpose of an embodiment of the present invention is to provide a device for farrowing judgment based on behavior recognition, which receives real-time overhead camera images, detects the standing / lying / farrowing state of a sow scheduled to farrow using DCNN-based technology, and when a farrowing event of a sow determined to be in a lying state is detected, a video recognition module is used to ultimately determine whether the detected farrowing event is an actual farrowing of a piglet or a false detection event due to the surrounding environment, and a method for operating the device.

[0006] A device for judging parturition based on behavior recognition of a sow according to an embodiment of the present invention includes a communication interface unit that receives a video of a sow preparing for parturition (sow), and a control unit that analyzes the received video and, when it is determined that the sow is in a lying state, determines whether the sow is actually parturient based on the result of the recognition of the parturition behavior of the sow and whether there is a false detection due to the surrounding environment.

[0007] The above control unit can determine the state of the sow according to the time change in the above-mentioned filmed image, calculate the probability value of standing and lying down, and determine the state of the sow based on the calculated probability value.

[0008] The above control unit can determine the state of the sow as a standing state when the standing state is maintained for a preset period of time when determining the standing state of the sow.

[0009] The above control unit can obtain a specified number of video frames before and after a video frame in which it is determined that parturition has been detected in order to determine whether the sow has actually parturinated, extract a region of interest (ROI), and make a final judgment on the actual parturition status based on the behavior recognition results of the extracted regions of interest.

[0010] The above control unit may use a video recognition-based classifier to generate the action recognition result.

[0011] The above control unit can calculate attention for the acquired previous and subsequent multiple video frames using a video recognition-based classifier and make a final judgment on the farrowing status of the sow based on the calculation result.

[0012] In addition, a method for operating a device for judging parturition based on behavior recognition of a sow according to an embodiment of the present invention includes a step of receiving a photographed image of a sow preparing for parturition by a communication interface unit, and a step of determining whether the sow is actually parturient based on a result of recognizing the parturition behavior of the sow by analyzing the received photographed image and determining whether the sow is in a lying state based on a result of recognizing the parturition behavior of the sow, based on a result of recognizing the parturition behavior of the sow.

[0013] The above-determining step can determine the state of the sow according to the change in time within the filmed image, calculate the probability value of standing and lying down, and determine the state of the sow based on the calculated probability value.

[0014] The above-determining step can determine the state of the sow as a standing state when the standing state is maintained for a preset period of time when determining the standing state of the sow.

[0015] The above-determining step may be performed by acquiring a specified number of video frames before and after a video frame in which it is determined that parturition has been detected in order to determine whether the sow has actually parturinated, extracting a region of interest (ROI) from each frame, and finally determining the actual parturition status based on the behavior recognition results of each of the extracted regions of interest.

[0016] The above-mentioned judging step may use a video recognition-based classifier to generate the above-mentioned action recognition result.

[0017] The above-determining step can be performed by calculating attention for the acquired previous and subsequent multiple video frames using the video recognition-based classifier, and making a final judgment on the farrowing status of the sow based on the calculation result.

[0018] According to an embodiment of the present invention, by installing an overhead camera in each farrowing crate where a sow is located, monitoring of the sow is performed, and at the same time, detection of a farrowing event during farrowing enables an immediate response by a worker such as a farrowing manager.

[0019] In addition, according to an embodiment of the present invention, the state of the sow can be confirmed by detecting the standing / lying state of the sow located in each farrowing crate.

[0020] Furthermore, according to an embodiment of the present invention, when a delivery event occurs, the final determination of whether or not the event is actually delivered can be made, thereby reducing false detections and increasing the efficiency of workers.

[0021] Figure 1 is a system for judging parturition based on behavior recognition of a sow according to an embodiment of the present invention.

[0022] Figure 2 is a block diagram illustrating the detailed structure of the model state analysis device or control device of Figure 1.

[0023] Figure 3 is a block diagram illustrating the detailed structure of the childbirth behavior recognition unit of Figure 2.

[0024] Figure 4 is a flowchart of a process for detecting a standing state of a mother pig according to an embodiment of the present invention.

[0025] FIG. 5 is a flowchart showing the parturition behavior recognition process of a device for parturition judgment based on behavior recognition of a sow according to an embodiment of the present invention; and

[0026] Fig. 6 is a flowchart showing another driving process of the model state analysis device or control device of Fig. 1.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0028] FIG. 1 is a diagram showing a farrowing judgment system based on behavior recognition of a sow according to an embodiment of the present invention.

[0029] As illustrated in FIG. 1, the farrowing judgment system (90) based on behavior recognition of a sow according to an embodiment of the present invention includes part or all of a photographing device (100), a sow condition analysis device (110), a worker terminal device (120), a communication network (130), and a control device (140).

[0030] Here, “including some or all” means that some components such as the control device (140) are omitted so that the sod condition analysis device (110) and the worker terminal device (120) can perform direct communication (e.g., P2P, etc.), or some or all of the components constituting the control device (140) can be integrated into a network device (e.g., wireless switching device, etc.) constituting a communication network (130), etc. In order to help a sufficient understanding of the invention, it is explained as including all.

[0031] The filming device (100) includes a camera that is installed in a farrowing crate, such as a sow preparing for farrowing, and performs a filming operation. Of course, various types of cameras, such as IP cameras and artificial intelligence cameras that can generate red, green, and blue RGB images, may be used here. The artificial intelligence camera may be configured to include an artificial intelligence module (e.g., hardware, software resources, etc.) for analyzing the captured images by the camera through an artificial intelligence program. In addition to the simple filming operation, the filming device (100) may also be configured as a device for analyzing the captured images. For example, the image analysis module may be mounted on the camera, but it may also be possible to use it by connecting it to a sow condition analysis device (110) in the form of an edge device, as shown in FIG. 1. Multiple cameras may be installed in the farrowing crate depending on the farm's situation, and all camera images may be provided to the sow condition analysis device (110).

[0032] The sow condition analysis device (110) is an image analysis device for analyzing the captured images provided by the photographing device (100). The sow condition analysis device (110) can be operated by being connected to a DB (110a), and the data of the captured images processed by the sow condition analysis device (110) can be stored and managed in the DB (110a). Of course, the DB (110a) can be replaced with a large-capacity memory installed inside the sow condition analysis device (110), so the embodiment of the present invention will not be particularly limited to the use of the DB (110a). The sow condition analysis device (110) can be equipped with an artificial intelligence module for image analysis, that is, a component configured by hardware or software or a combination thereof, and the sow condition analysis device (110) according to the embodiment of the present invention can operate as at least one device among devices that perform a farrowing judgment operation based on behavior recognition of a sow. Here, "operating as at least one device" means that the sow condition analysis device (110) independently performs all operations related to sow farrowing. Alternatively, it may mean that the sow condition analysis device (110) can operate in conjunction with the control device (140) of FIG. 1 to perform operations for determining sow farrowing.

[0033] For example, when the sow condition analysis device (110) processes all movements independently, it can receive the captured images from the shooting device (100), analyze the captured images according to the embodiment of the present invention, and determine the standing and lying down states of the sow. Then, if it is determined to be in the lying down state, the sow's behavior recognition model, i.e., program, is applied to make a final determination as to whether or not actual delivery has occurred. In other words, event information related to the sow's delivery can be preset in order to make a final determination as to whether or not actual delivery has occurred. Therefore, if the result of the sow's behavior recognition falls within the category of the preset event information, it can determine that the sow is in delivery and operate to notify the operator terminal device (120) of FIG. 1 of related information. Of course, the sow condition analysis device (110) can determine whether the pre-registered operator terminal device (120) is within a communication range, and when it is determined as being within the range, it can notify the sow of the delivery state through direct communication. Here, direct communication may mean communication such as Bluetooth, but may also mean communication based on the Internet of Things (IoT), and thus the embodiment of the present invention will not be particularly limited to any one form.

[0034] For example, let's assume that the worker terminal (120) is out of the communication radius where the sow condition analysis device (110) can communicate. In this case, information related to the sow's farrowing status can be notified to the worker terminal (120) via a wireless switching device or gateway constituting the communication network (130). The sow condition analysis device (110) requests the wireless switching device of the communication network (130) to notify the worker terminal (120) of the sow's farrowing status. Accordingly, the wireless switching device can notify the worker terminal (120) of the message requested by the sow condition analysis device (110), i.e., the sow's farrowing status. Of course, the notification can be in the form of a voice call such as ARS, and notification of preset content can also be made via a text message (SMS) or messenger-based chat service. Of course, in relation to such notifications, it is entirely possible to operate in conjunction with the control device (140) of Fig. 1, so the embodiment of the present invention will not be particularly limited to any one form.

[0035] Based on the above, it may be desirable for the sow condition analysis device (110) according to the embodiment of the present invention to operate in conjunction with a control device (140), etc. The sow condition analysis device (110) preferably directly processes all operations related to the determination of sow farrowing in order to prevent distortion of analysis data of captured images and to prevent network load on the communication network (130). In addition, the sow condition analysis device (110) may provide related data to the control device (140) so that it may be stored and managed.

[0036] The sow condition analysis device (110) can receive surveillance camera images of sows from the filming device (100), preprocess them, and store the necessary (video) frames. Then, the preprocessed images can be used to determine whether the sow is standing or farrowing. Furthermore, the sow condition analysis device (110) can transmit the results of the determination to the control device (140) via the communication network (130) of FIG. 1 so that a user, such as a farrowing manager, can be informed of the farrowing status of the sow.

[0037] As will be discussed in more detail later, the sow condition analysis device (110) according to an embodiment of the present invention can adjust the size of the input image for reasons such as improving the processing speed and perform image quality improvement tasks such as normalization for preprocessing operations. In addition, it receives an input image using a DCNN trained to determine whether a sow is standing and outputs the probability that it is in a standing or lying state. The probability value can be output based on a series of input images input for a specified period of time. The sow condition analysis device (110) can determine the standing state by applying a standing detection DCNN and determine the other states as a lying state. The sow condition analysis device (110) performs an operation to determine whether the sow, i.e., the sow object in the image, is in a lying state. The sow condition analysis device (110) determines the state of the sow based on a value with a higher probability between standing and lying down. In addition, the sow condition analysis device (110) determines whether the detected sow exists within the event area. Here, the event area is an area set by the user interface, which may mean an area set by specifying an area in the image where the sow should be detected. For example, if the sow remains in a standing state for T seconds, the sow's state is updated (or determined) to a standing state, and the corresponding information is transmitted to the control device (140) to display the sow's state.

[0038] When the sow condition analysis device (110) detects parturition, it extracts a certain area (or region of interest) (ROI) based on the center of the detection box of a certain number of video frames (or input images) before and after the corresponding video frame in which parturition is determined to have been detected. Then, the video frames from which the ROI has been extracted go through a video sequence generation process and are input to a video action recognition module, and parturition can be finally determined by a classifier based on the video recognition module. The video recognition module according to an embodiment of the present invention may be composed of two elements, an encoder and a decoder, and the encoder (or visual information extraction unit) obtains parturition event video data and extracts important visual information of the video using it, and the decoder (or action prediction unit) uses the extracted information to predict a specific action within the video. Since the module calculates attention for multiple frames of the video input and infers information for each frame, it has the advantage of being more accurate than existing detectors that detect objects with only one frame. By utilizing these advantages, a classifier trained by true detection and false detection clips compares a certain number of frames before and after the birth event detection time of the existing birth detector. If the result of the comparison determination is not a birth, a result indicating that the event is not an actual birth is transmitted. Conversely, if it is determined that a birth actually occurred, the sow object status is updated to a birthing status and the result is transmitted to the control device (140). Through this, information related to the birthing status of the sow can be notified to the work terminal device (120).

[0039] Attention refers to the task of calculating similarity by utilizing the hidden state vector information of "all time points" of the encoder to find the input (source) part with high similarity at each time point of the decoder. It can be understood as transmitting the hidden state information of all time points of the encoder regarding the similarity with the hidden state of the decoder. The hidden state vector is a concept used in sequence models such as recurrent neural networks (RNNs), and refers to a vector representing the internal state of the model that reflects the sequence information input up to the present time. In a sequence model, input is received at each time point (time step), and the hidden state vector of the previous time point is passed to the model along with the input of the current time point to calculate the hidden state vector of the current time point. The hidden state vector thus calculated is then used to calculate the input of the next time point, and if this process is repeated until the end of the sequence, the hidden state vector for all inputs is calculated.

[0040] In summary, the sow condition analysis device (110) according to an embodiment of the present invention determines and detects the standing state of a sow into two states: lying down and standing up. The state in which the sow stands up to get feed / drink water is determined as the standing state, and the state in which the sow sits or lies on the side is determined as the lying down state, and the corresponding states are detected as standing / lying down events, respectively. In the case of the sow standing up, whether or not the sow has given birth is not determined, but in the case of the sow lying down, whether or not the sow has given birth is determined and detected as an event. At this time, the behavior recognition model-based classification system ultimately determines whether the detected giving birth event is a false positive caused by the surrounding environment or an actual giving birth.

[0041] The worker terminal (120) may include a terminal device, such as a smartphone, carried by a farrowing manager who observes and manages the farrowing status of a sow in a farrowing crate, or a manager who raises pigs. Of course, the worker terminal (120) may include any type of device other than a smartphone as long as it can quickly contact the farrowing manager or raising manager. The worker terminal (120) may perform direct communication with the sow condition analysis device (110) or receive notification of the status of the sow in the farrowing crate via the communication network (130) or control device (140) of FIG. 1. More precisely, the worker terminal (120) is intended to address the problem of missing the golden time necessary for piglet care due to farrowing, thereby lowering the piglet survival rate, by receiving notification when a sow is predicted to farrow soon or is in the process of farrowing. Therefore, the worker terminal (120) can be viewed as notifying a farrowing manager or the like of information related to the prediction when a sow is predicted to farrow soon so that the farrowing manager can prepare for it. Of course, the worker terminal (120) may also receive information corresponding to its location based on the notification from the control device (140). Typically, when a farrowing manager, etc. is determined to be located not far from the farrowing crate, the relevant information is notified 5 minutes prior to the sow's farrowing. However, when the farrowing manager, etc. is determined to be located within 30 minutes (e.g., on foot, by public transportation, etc.), the relevant information may be notified taking that time into account. The control device (140) of Fig. 1 may be involved in such operations.

[0042] The communication network (130) includes both wired and wireless communication networks. For example, a wired or wireless Internet network may be used or linked as the communication network (130). Here, the wired network includes an Internet network such as a cable network or a public switched telephone network (PSTN), and the wireless communication network includes a CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), WiBro network, etc. Of course, the communication network (130) according to the embodiment of the present invention is not limited thereto, and may be used as an access network of a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network in a cloud computing environment, a 5G network, etc. For example, if the communication network (130) is a wired communication network, an access point within the communication network can connect to a telephone exchange, etc., but if it is a wireless communication network, data can be processed by connecting to an SGSN or GGSN (Gateway GPRS Support Node) operated by a communication company, or data can be processed by connecting to various relays such as a BTS (Base Transceiver Station), NodeB, or e-NodeB.

[0043] The communication network (130) may include an access point. The access point includes a small base station such as a femto or pico base station, which is often installed inside a building. Here, the femto or pico base station is classified according to the maximum number of devices such as a sow condition analysis device (110) or a worker terminal device (120) that can be connected to the small base station. Of course, the access point may include a short-range communication module for performing short-range communication such as Zigbee and Wi-Fi with the sow condition analysis device (110) or the worker terminal device (120). The access point may use TCP / IP or RTSP (Real-Time Streaming Protocol) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, Zigbee, infrared (IrDA), radio frequency (RF) such as UHF (Ultra High Frequency) and VHF (Very High Frequency) in addition to Wi-Fi, and ultra-wideband communication (UWB). Accordingly, the access point can extract the location of the data packet, designate the best communication path for the extracted location, and forward the data packet along the designated communication path to the next device, such as the control device (140). The access point can share multiple lines in a typical network environment, and includes, for example, routers, repeaters, and repeaters.

[0044] The control device (140) can operate as at least one device among the devices for determining farrowing based on behavior recognition of a sow according to an embodiment of the present invention. Here, the at least one device means that the control device (140) can perform any operation such as the sow condition analysis device (110) of FIG. 1. Of course, since the control device (140) can be configured as a server, etc., it is not difficult to perform the operation of the sow condition analysis device (110), and since the server is expensive, it is highly likely that the hardware resources or software resources will be higher in performance than the sow condition analysis device (110). Accordingly, it is possible to analyze the sow condition by receiving a video related to the sow's farrowing condition from the filming device (100), but considering the network load of the communication network (130), it is preferable that the sow condition analysis operation be analyzed through the sow condition analysis device (110) configured on the side of the filming device (100). However, the control device (140) may periodically receive analysis data from the sow condition analysis device (110) and store and manage it.

[0045] In addition, if the control device (140) must analyze the captured images of the photographing devices (100) installed in multiple farrowing crates, the data processing burden may increase. Therefore, since the computational processing burden increases, the control device (140) preferably receives the analysis result data from the sow condition analysis device (110), stores and manages it in the DB (140a) of FIG. 1, and further operates to notify the worker terminal device (120), such as the farrowing manager who manages farrowing for sows in a specific farrowing crate, of the status related to the farrowing of the sow. Of course, the control device (140) may also be able to install and operate the sow condition analysis device (110) only in a limited or selective manner for the photographing devices (100) in an area where network load is a concern. Therefore, the embodiment of the present invention will not be particularly limited to any one form.

[0046] For example, if a false detection is made despite being searched (or analyzed) through farrowing behavior recognition in the actual sow condition analysis device (110), the control device (140) may receive the relevant image, relearn the related image, and transmit the learning result to the sow condition analysis device (110). In other words, since the embodiment of the present invention prevents false detection by checking the farrowing status image searched by the farrowing behavior recognition unit of the sow condition analysis device (110) for some of the image frames when a farrowing event is detected primarily due to many false detections of farrowing events, the sow condition analysis device (110) and the control device (140) can work together to (re)learn the image and transmit the result. The sow condition analysis device (110) or the farrowing behavior recognition unit of the control device (140) can learn and input (or set) false detection factors such as previously learned feces.

[0047] In summary, the control device (140) according to the embodiment of the present invention can perform various operations. First, it can perform all operations related to farrowing in the sow condition analysis device (110) of FIG. 1. In other words, it receives a photographed image of the sow from the photographing device (100) or the sow condition analysis device (110), determines the standing / lying state of the sow, and performs an operation to determine whether or not farrowing is actually occurring when the sow is in the lying state, and notifies the operator terminal (120) of the fact when farrowing is predicted. Second, the control device (140) can perform some operations in conjunction with the sow condition analysis device (110). For example, the sow condition analysis device (110) can receive only the data of video frames that are at risk of false detection, perform an analysis operation, and then provide the results to the sow condition analysis device (110). Third, the control device (140) may simply store and manage the analysis data of the sow condition analysis device (110), provide analysis results when data is requested from the worker terminal device (120), and perform an operation to notify the worker terminal device (120) of the fact of sow farrowing at the request of the sow condition analysis device (110). Only passive operations are performed.

[0048] According to the above configuration, an overhead camera is installed in the farrowing crate of a sow farrowing pen, and the received images are monitored and analyzed in real time to check the standing / lying state of the sow. Here, each crate where a sow enters the pen one week before the expected farrowing and stays for about two weeks after farrowing is called a farrowing crate, and the farrowing pen refers to a place where these farrowing crates are gathered. In other words, farrowing crates are located within the farrowing pen. In addition, by determining whether the farrowing detected is an actual farrowing or a false detection due to the surrounding environment, the work efficiency of workers and the survival rate of piglets can be improved.

[0049] In addition, according to the above configuration, the fatigue of workers and the decrease in the survival rate of piglets caused by the inability to provide real-time status monitoring of numerous sows and immediate care for simultaneous farrowing in a farrowing barn are resolved by utilizing DCNN-based object detection and recognition technology to detect the standing / lying state of sows and to make a final judgment on the actual occurrence of farrowing through a classification system based on a behavior recognition model, thereby performing efficient monitoring and improving survivability, thereby increasing the work efficiency of workers and the economic profits of farms.

[0050] In addition to the above, the photographing device (100), the sow condition analysis device (110), the worker terminal device (120), the communication network (130), and the control device (140) of FIG. 1 can perform various operations, and other detailed information can be covered in more detail later, so we would like to replace it with those contents.

[0051] Fig. 2 is a block diagram illustrating a detailed structure of the sow condition analysis device or control device of Fig. 1, and Fig. 3 is a block diagram illustrating a detailed structure of the farrowing behavior recognition unit of Fig. 2.

[0052] As illustrated in FIG. 2, the sow condition analysis device (110) or control device (140) of FIG. 1 (hereinafter, referred to as the sow condition analysis device) according to an embodiment of the present invention includes part or all of a communication interface unit (200), a control unit (210), a farrowing behavior recognition unit (220), and a storage unit (230).

[0053] Here, "including some or all" means that some components, such as a storage unit (230), are omitted and a device (110 / 140) for all of the sow standing state analysis, farrowing, and farrowing judgment based on sow behavior recognition, such as a sow state analysis device (110), is configured, or some components, such as a farrowing behavior recognition unit (220), can be integrated into other components, such as a control unit (210), and so on. In order to help a sufficient understanding of the invention, it is described as including all.

[0054] The communication interface unit (200) communicates with the control device (140) via the photographing device (100) of FIG. 1 and the communication network (130) of FIG. 1, respectively. The communication interface unit (200) performs operations such as modulation / demodulation, muxing / demuxing, encoding / decoding, and scaling to convert resolution during the communication process, which are obvious to those skilled in the art and therefore will not be further described.

[0055] The communication interface unit (200) can receive the captured images provided from the photographing device (100) via wired or wireless communication and transmit them to the control unit (210) for image analysis. In addition, the communication interface unit (200) can transmit event information related to image analysis to the control device (140) of FIG. 1 under the control of the control unit (210). Here, the event information can include information that the sow in the farrowing crate monitored by the photographing device (100) is currently predicted to farrow or is in the process of farrowing.

[0056] Furthermore, the communication interface unit (200) can periodically provide analysis data of the filmed images to the control device (140) of FIG. 1 under the control of the control unit (210) so that the data can be stored and managed in the DB (140a) of FIG. 1. For example, the control device (140) of FIG. 1 can analyze filmed images of sows installed in multiple farrowing crates to collect analysis data to create big data, and further analyze the big data through an artificial intelligence program to extract features related to the farrowing of the sows. Through this, the farrowing state of the sow can be learned, and the sow state analysis device (110) can operate to predict the farrowing state based on this. Through this, it may be possible to accurately predict the farrowing state, which may be somewhat different depending on the type of sow.

[0057] The control unit (210) is responsible for the overall control operations of the communication interface unit (200), the farrowing behavior recognition unit (220), and the storage unit (230) of FIG. 2. When the control unit (210) receives a photographed image from the photographing device (100) of FIG. 1 through the communication interface unit (200), the control unit (210) can temporarily store the image in the storage unit (230) and then provide the image to the farrowing behavior recognition unit (220) for image analysis. In addition, when the control unit (210) predicts that a sow will farrow at the request of the farrowing behavior recognition unit (220), the control unit (210) can search for a nearby worker terminal device (120), and, for example, when a worker terminal device (120) is searched within the communication radius, the control unit (210) can control the communication of the communication interface unit (200) to transmit a message to the worker terminal device (120) stating that the sow is likely to give birth soon. Of course, in this process, if the worker terminal (120) is located at a long distance, such as a delivery manager, it may be possible to notify the worker terminal (120) of the fact based on the contact information of the worker terminal (120) through the communication network (130) or control device (140) of FIG. 1.

[0058] In addition, the control unit (210) can perform various operations, but for example, let's assume that learning data that can determine the farrowing status related to the farrowing of a sow is provided by providing the analysis data of the filmed image to the control device (140) of FIG. 1. In other words, the control device (140) of FIG. 1 can accurately determine the farrowing status of a specific sow through big data analysis, and store the farrowing status data related thereto in the sow status analysis device (110) of FIG. 1 so that the farrowing status of the sow can be accurately recognized or determined based on this. Of course, in the embodiment of the present invention, when the farrowing status is assumed to be an event in order to determine the farrowing status of a sow, the event information can be preset in the sow status analysis device (110). For example, it is preset to predict and notify the farrowing status when a specific action occurs repeatedly or when a series of actions that occur over time exactly match. In order to accurately predict the farrowing status of a specific sow, various methods are possible, such as using data set based on a user interface or using analysis results based on big data analysis. Therefore, the present invention will not be specifically limited to any one method. The control unit (210) may be involved in such operations.

[0059] The farrowing behavior recognition unit (220) analyzes the video footage of the sow to determine whether the sow is standing or lying down. Furthermore, the farrowing behavior recognition unit (220) performs an operation to determine whether the farrowing actually occurred or was a false positive due to the surrounding environment when farrowing was detected. For example, the farrowing behavior recognition unit (220) utilizes DCNN-based object detection and recognition technology to detect the sow's standing / lying state, and determines whether the farrowing actually occurred using a classification system based on a behavior recognition model. As a result, efficient monitoring is performed and survivability is enhanced, thereby increasing worker efficiency and farm income.

[0060] The labor act recognition unit (220) may include, for example, a part or all of the image input unit (300), the standing detection and determination unit (310), the labor detection unit (320), and the labor act recognition and determination unit (330) as shown in FIG. 3. Here, “including a part or all” is not significantly different from the meaning above, so it will be replaced with those contents. The image input unit (300), the standing detection and determination unit (310), the labor detection unit (320), and the labor act recognition and determination unit (330) in FIG. 3 may be configured by hardware, software, or a combination thereof, and thus the present invention will not be particularly limited to any one form.

[0061] The image (data) input unit (300) receives an image acquired from a surveillance camera as input. At this time, the preprocessing unit included in the image input unit (300) or the standing detection and discrimination unit (310) adjusts the size of the input image for reasons such as improving the processing speed and performs image quality improvement tasks such as normalization. The standing detection and discrimination unit (310) can be equipped with a standing detection DCNN, and through this, the DCNN trained to determine whether the sow is standing can receive an input image and output the probability (value) of the standing or lying state. At this time, the standing detector of the standing detection and discrimination unit (310) can use the yolo series model, which has the best performance in real-time processing, for fast processing. The yolo model is a one-stage series model that performs detection and classification simultaneously, and has the advantage of being faster than a two-stage model that performs classification separately after object detection. The detector is trained by sample data of the standing and lying states of the sow. The class has two classes, standing and lying down, and performs an action to determine whether or not the object (sow) is giving birth when it is in a lying down state. Afterwards, the state of the sow is determined based on the value with a higher probability between standing and lying down. It is determined whether the detected sow exists within the event area. The event area is set in the user interface (screen) and is set by specifying the area in the image where the sow should be detected. If the sow is maintained in a standing state for T seconds, the state of the sow is updated to a standing state. The updated information is transmitted to the control device (140), and the control device (140) can display the state of the sow.

[0062] The farrowing detection unit (320) determines whether or not farrowing occurs when the object (sow) is in a lying state as described above, and can then determine the state of the sow based on a value with a higher probability between standing and lying down. In addition, the farrowing detection unit (320) can determine whether the detected sow exists within an event area. When farrowing is detected, a certain region of interest (ROI) is extracted based on the center of the detection box of a certain number of frames before and after the frame in which farrowing is determined to have been detected. Thereafter, the frames from which the ROI has been extracted go through a video sequence generation process and are sent as input to the video behavior recognition module, and farrowing is finally determined by a classifier based on a video recognition module of the farrowing behavior recognition determination unit (330) that operates as a video behavior recognition module.

[0063] The video recognition module utilized in the classifier consists of two components, an encoder and a decoder, as described above. The encoder fetches parturition event video data and extracts important visual information from the video, and the decoder uses the extracted information to make predictions about specific actions within the video. This module calculates attention for multiple frames of the video input and infers information for each frame, making it more accurate than existing detectors that detect objects with only a single frame. Utilizing this advantage, the existing parturition detector can detect parturition events at a certain number of frames before and after the time of detection by the existing parturition detector. If the comparison result is determined to be not parturition through a classifier trained on true and false positive clips, a result indicating that the event is not an actual parturition can be transmitted to the control device (140) of FIG. 1. Conversely, if the parturition is determined to have actually occurred, the sow object status can be updated to a parturition status and the result can be transmitted to the control device (140).

[0064] The storage unit (230) of FIG. 2 can temporarily store various types of data processed under the control of the control unit (210). For example, when a captured image of the photographing device (100) of FIG. 1 is received, it can be temporarily stored and then provided to the labor action recognition unit (220) for image analysis.

[0065] In addition to the above, the communication interface unit (200), control unit (210), labor behavior recognition unit (220), and storage unit (230) of FIG. 2 can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with those contents.

[0066] According to an embodiment of the present invention, the communication interface unit (200), the control unit (210), the labor behavior recognition unit (220), and the storage unit (230) of FIG. 2 are configured as physically separate hardware modules, but each module may store and execute software for performing the above operations. However, the software is a collection of software modules, and each module may be formed of hardware, so there is no particular limitation on the configuration such as software or hardware. For example, the storage unit (230) may be hardware such as storage or memory. However, since it is also possible to store information (repository) in software, there is no particular limitation on the above.

[0067] Meanwhile, as another embodiment of the present invention, the control unit (210) may include a CPU and a memory, and may be formed as a single chip. The CPU may include a control circuit, an operation unit (ALU), a command interpretation unit, and a registry, and the memory may include a RAM. The control circuit may perform a control operation, the operation unit may perform an operation of binary bit information, and the command interpretation unit may perform an operation of converting a high-level language into machine language and vice versa, including an interpreter or a compiler, and the registry may be involved in software data storage. According to the above configuration, for example, at the initial operation of the sow condition analysis device (110), the program stored in the farrowing behavior recognition unit (220) may be copied and loaded into a memory device, and then executed, thereby rapidly increasing the data operation processing speed. In the case of a deep learning model, it may be loaded onto a calculation execution device such as a GPU instead of a RAM, and the execution speed may be accelerated and executed using the GPU.

[0068] Figure 4 is a flowchart of a process for detecting a standing state of a mother pig according to an embodiment of the present invention.

[0069] For convenience of explanation, FIG. 4 will be described together with the sow condition analysis device (110) of FIG. 1. The sow condition analysis device (110) according to the embodiment of the present invention receives an image from the photographing device (100) (S400).

[0070] Furthermore, through preprocessing units, the input image can be resized and image quality can be improved through tasks such as standardization to improve processing speed (S410). Image quality improvement can be achieved through various methods, such as interpolation, which corrects the pixel value of a specific pixel using surrounding pixels.

[0071] Next, the sow state analysis device (110) receives an input image as a DCNN (or from a DCNN) that has been trained to determine whether the sow is standing up through a DCNN for standing detection, and outputs the probability of the sow being in a standing or lying down state (S420). It can also be seen as generating a probability value. At this time, the standing detector can use a yolo series model that has the best performance in real-time processing for fast processing. The detector is trained by sample data of standing and lying down states. There are two classes, standing and lying down, and it can perform an operation of determining whether or not to give birth when the object (sow) is in a lying down state.

[0072] Afterwards, the sow condition analysis device (110) determines the sow condition based on the value with the greater probability of standing or lying down (S430). In other words, if the probability of lying down is high, and if the probability of lying down is also large, the sow condition is in the lying down state.

[0073] And the sow condition analysis device (110) determines whether the detected sow exists within the event area (S440). The event area can be set by specifying the area in which the sow should be detected in the image through the user interface or by setting it on the interface screen.

[0074] In addition, the sow condition analysis device (110) updates the sow condition to a standing condition if it maintains a standing condition for T seconds as a result of the above judgment (S450, S460).

[0075] Next, the modon status analysis device (110) can transmit the updated information to the control device (140), and the control device (140) can operate to display the modon status (S470).

[0076] In addition to the above, detailed information related to the model state analysis device (110) of Fig. 1 has been sufficiently explained above, so we would like to replace it with that information.

[0077] Figure 5 is a flowchart showing the farrowing behavior recognition process of a device for farrowing judgment based on behavior recognition of a sow according to an embodiment of the present invention.

[0078] For convenience of explanation, if Fig. 5 is described together with the sow condition analysis device (110) of Fig. 1, the sow condition analysis device (110) receives an image from a surveillance camera, analyzes the image, and performs a farrowing detection operation (S500, S510).

[0079] At this time, when the sow condition analysis device (110) detects parturition, it extracts a certain area (ROI) based on the center of the detection box of a certain number of frames before and after the frame in which parturition is determined to have been detected (S520).

[0080] Afterwards, the sow condition analysis device (110) sends the frames from which the ROI has been extracted as input to the video action recognition module through a video sequence generation process, and the classifier based on the video recognition module finally determines parturition (S530 to S550). The video recognition module utilized in the system consists of two elements: an encoder and a decoder. The encoder fetches parturition event video data and extracts important visual information from the video, and the decoder fetches the extracted information and performs a prediction for a specific action within the video. Since the module calculates attention for multiple frames of the video input and infers information for each frame, it is more accurate than existing detectors that detect objects with only one frame.

[0081] By utilizing these advantages, the sow condition analysis device (110) compares a certain number of frames before and after the time point of detection of a farrowing event by an existing farrowing detector through a classifier learned by true detection and false detection clips, and if it determines that the event is not a real farrowing, it transmits a result indicating that the event is not a real farrowing (S560). Conversely, if it is determined that a farrowing has actually occurred, it updates the sow object status to a farrowing status and transmits the result to the control device (140) (S570).

[0082] In addition to the above, the model state analysis device (110) of FIG. 1 can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with that information.

[0083] Fig. 6 is a flowchart showing another driving process of the model state analysis device or control device of Fig. 1.

[0084] For convenience of explanation, referring to FIG. 6 together with the sow condition analysis device (110) of FIG. 1, the sow condition analysis device (110) according to an embodiment of the present invention receives a photographed image of a sow preparing for farrowing (S600). For precise analysis, an RGB image with good resolution or a resolution higher than a standard value may be used as the photographed image.

[0085] In addition, the sow state analysis device (110) analyzes the received video and, when it is determined that the sow is in a lying down state, it applies a technology for recognizing the sow's farrowing behavior, such as a technology for calculating attention using a certain number of video frames before and after the time point determined to be farrowing and predicting farrowing based on the calculation result, to determine whether the sow is actually farrowing or not from false detection due to the surrounding environment (S610). In the video, the sow can be referred to as an object. Therefore, it is not difficult to determine the standing state and lying state of the sow by analyzing the video, that is, analyzing the characteristics of the video and making a judgment based on the result.

[0086] However, it may be possible to determine whether a sow lying down is actually in labor through a process similar to that in Figure 5. For example, a video behavior recognition module, i.e., a program, can make a final determination of labor using a classifier based on a video recognition module. To this end, the video recognition module may include an encoder and a decoder, which can be used to calculate attention for input video frames and, based on the calculation results, infer information for each video frame to make a final determination of labor.

[0087] In addition to the above, the model state analysis device (110) of FIG. 1 can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with that information.

[0088] Meanwhile, even though all components constituting the embodiments of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present invention, all of the components may be selectively combined and operated one or more times. In addition, although all of the components may be implemented as individual independent hardware, some or all of the components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions of the combined hardware in one or more pieces. The codes and code segments constituting the computer program can be easily inferred by those skilled in the art of the present invention. Such a computer program may be stored in a non-transitory computer-readable storage medium and read and executed by a computer, thereby implementing the embodiments of the present invention.

[0089] Here, the non-transitory readable storage medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the above-described programs may be stored and provided on a non-transitory readable storage medium, such as a CD, DVD, hard disk, Blu-ray disc, USB, memory card, or ROM.

[0090] Although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art without departing from the spirit or scope of the present invention as claimed in the claims. Furthermore, such modifications should not be understood individually from the technical idea or prospect of the present invention.

[0091] ** Explanation of symbols

[0092] 100: Filming device 110: Sow condition analysis device

[0093] 120: Worker terminal 130: Communication network

[0094] 140: Control device 200: Communication interface unit

[0095] 210: Control unit 220: Childbirth behavior recognition unit

[0096] 230: Storage unit 300: Video input unit

[0097] 310: Standing detection and determination unit 320: Labor detection unit

[0098] 330: Childbirth Behavior Recognition and Discrimination Unit

Claims

1. A communication interface unit that receives a video of a sow preparing for delivery; and A control unit that analyzes the received video footage and determines whether the sow is in a lying state and whether the sow is actually giving birth based on the recognition result of the sow's giving birth behavior; A device for judging parturition based on behavioral recognition of a mother pig, including:

2. In paragraph 1, The above control unit is a device for judging parturition based on behavior recognition of a sow, which judges the state of the sow according to the time change in the above-mentioned filming image, calculates the probability of standing and lying down, and judges the state of the sow based on the calculated probability value.

3. In paragraph 2, The above control unit is a device for judging farrowing based on behavior recognition of a sow, which determines the state of the sow as a standing state when the standing state is maintained for a preset period of time when judging the standing state of the sow.

4. In paragraph 2, The above control unit is a device for judging farrowing based on behavior recognition of a sow, wherein the control unit acquires a specified number of video frames before and after a video frame in which farrowing is determined to have been detected in order to determine whether the sow has actually farrowed, extracts a region of interest (ROI), and finally determines the actual farrowing state based on the behavior recognition results of the extracted regions of interest.

5. In paragraph 4, The above control unit is a device for judging farrowing based on behavior recognition of a sow, which uses a video recognition-based classifier to generate the behavior recognition result.

6. In paragraph 5, The above control unit is a device for judging farrowing based on behavior recognition of a sow, which calculates attention for a plurality of previous and subsequent video frames acquired using a classifier based on video recognition and makes a final judgment on the farrowing status of the sow based on the calculation result.

7. A step in which the communication interface unit receives a video of a sow preparing for parturition; and A step of the control unit analyzing the received video footage and determining whether the sow is actually giving birth based on the recognition result of the sow's giving birth behavior and whether there is a false detection due to the surrounding environment when the sow is determined to be in a lying state; A method for operating a device for judging parturition based on behavioral recognition of a sow, including:

8. In paragraph 7, The above judging step is, A method for operating a device for judging farrowing based on behavior recognition of a sow, which judges the state of the sow according to changes in time within the above-mentioned video, calculates the probability of standing and lying down, and judges the state of the sow based on the calculated probability value.

9. In paragraph 8, The above judging step is, A method for operating a device for judging farrowing based on behavior recognition of a sow, wherein the standing state of the sow is determined to be a standing state when the standing state is maintained for a preset period of time when judging the standing state of the sow.

10. In paragraph 8, The above judging step is, A method for operating a device for judging parturition based on behavior recognition of a sow, wherein a specified number of video frames before and after a video frame in which parturition is determined to have been detected are acquired to determine whether the sow has actually parturinated, and a region of interest (ROI) is extracted from each of the video frames, and a final judgment is made as to the actual parturition status based on the behavior recognition results of each of the extracted regions of interest.

11. In paragraph 10, The above judging step is, A method for operating a device for judging farrowing based on behavior recognition of a sow, using a video recognition-based classifier to generate the above behavior recognition results.

12. In paragraph 11, The above judging step is, A method for operating a device for judging farrowing based on behavior recognition of a sow, wherein attention is calculated for a plurality of previous and subsequent video frames acquired using the above-described video recognition-based classifier, and a final judgment is made on the farrowing status of the sow based on the calculation result.

Citation Information

Patent Citations

  • Delivery timing notification system and delivery timing notification method

    JP2021121176A

  • Non-conductive film and preparation method for semiconductor laminate

    KR1020200143289A

  • Sow Management Device, System and Method based on AI Processor

    KR102302040B1

  • Image-based sow farrowing notification system

    KR102372107B1

  • Apparatus for Judging Farrowing Based on Sow's Behavior Recognition and Driving Method Thereof

    KR102600622B1