A method and device for monitoring livestock delivery behavior

Abstract

The present invention provides a method and device for monitoring livestock delivery behavior, the method comprising: obtaining a thermal infrared image and a color image; inputting the color image into a target detection model to obtain a cub identification result; if it is determined that there is a cub in the livestock hip area according to the cub identification result, then obtaining the region coordinates of the cub target frame in the color image; converting the region coordinates into the region coordinates on the thermal infrared image to determine the cub target frame region in the thermal infrared image; and determining the livestock delivery behavior judgment result according to the temperature distribution in the cub target frame region. The method and device for monitoring livestock delivery behavior provided by the present invention adopts a multi-data fusion technology of color images and thermal infrared images, monitors livestock delivery behavior by fusing temperature and target motion detection, has a high recognition accuracy, and effectively improves the automation level of livestock breeding.

Description

Technical Field

[0001] The invention relates to the field of animal husbandry, and in particular to a method and a device for monitoring the birthing behavior of livestock. Background Art

[0002] At present, in the animal husbandry industry, the livestock delivery and breeding stages require a large investment of manpower and material resources.

[0003] Taking pig farming as an example, it depends on multiple pig farm managers to inspect the sow pens and deliver piglets in time. Especially in winter, if piglets are not found in time, they are prone to suffocation and freezing to death, causing great economic losses to the pig farm.

[0004] In particular, in medium and large pig farms under the mainstream batch management model, sows give birth in the same delivery room basically within 1 to 2 days. During the sow delivery management work, it is difficult for managers to judge whether the sows are experiencing dystocia.

[0005] At present, relevant intelligent equipment and systems have not been applied in the livestock delivery process in China. Against this background, intelligent real-time monitoring methods and devices for livestock delivery are urgently needed to be developed. Summary of the invention

[0006] The present invention provides a method and device for monitoring livestock delivery behavior, which are used to solve the defects of the prior art that livestock delivery behavior needs to be monitored manually, which is time-consuming and labor-intensive, and realizes the automated management of livestock delivery behavior monitoring.

[0007] In a first aspect, the present invention provides a method for monitoring livestock parturition behavior, comprising:

[0008] Acquire thermal infrared images and color images containing the target livestock's rump area;

[0009] Inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image;

[0010] If it is determined according to the cub identification result that there is a cub in the buttocks area of ​​the target livestock, obtaining the first area coordinates of the cub target frame corresponding to the cub in the color image;

[0011] Converting the first area coordinates into second area coordinates on the thermal infrared image to determine a cub target frame area of ​​the cub in the thermal infrared image;

[0012] The result of judging the livestock's parturition behavior is determined according to the temperature distribution in the target frame area of ​​the young animal.

[0013] According to a livestock delivery behavior monitoring method provided by the present invention, the livestock delivery behavior judgment result is determined according to the temperature distribution in the cub target frame area, including:

[0014] If the highest temperature in the cub target frame area is greater than the temperature threshold of the cub just born, the livestock birthing behavior judgment result is determined to be birthing, otherwise the livestock birthing behavior judgment result is determined to be other birthing states.

[0015] According to a livestock delivery behavior monitoring method provided by the present invention, the livestock delivery behavior judgment result is determined according to the temperature distribution in the cub target frame area, including:

[0016] Calculate the median temperature in the cub target frame area as the average temperature of the cub target frame area;

[0017] Determine a surrounding area within a preset range outside the cub target frame area, and calculate the median temperature of the surrounding area as the average temperature of the surrounding area;

[0018] If the difference between the average temperature of the target area of ​​the young piglet and the average temperature of the surrounding area is greater than a preset temperature difference threshold, the livestock delivery behavior judgment result is determined to be delivery; otherwise, the livestock delivery behavior judgment result is determined to be other delivery states.

[0019] According to a method for monitoring livestock delivery behavior provided by the present invention, after determining that the livestock delivery behavior judgment result is other delivery states, the method further includes:

[0020] Acquire a plurality of continuous color image frames within a preset time length related to the color image;

[0021] Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and determining the cub target frame corresponding to each frame of color image according to the cub recognition result corresponding to each frame of color image output by the target detection model;

[0022] Using the target tracking model to track all piglet target frames, and obtaining the activity trajectory of the piglets within the preset time period;

[0023] If it is determined according to the activity trajectory that the cub continues to move within the preset time period, then the other delivery state is determined to be completion of delivery;

[0024] If it is determined according to the activity trajectory that the cub is stationary within the preset time period, and the highest temperature in the cub target frame area is less than the dead fetus temperature threshold, then the other delivery state is determined to be a dead fetus.

[0025] According to a livestock delivery behavior monitoring method provided by the present invention, if it is determined that there is no cub in the buttocks area of ​​the target livestock according to the cub identification result, the method further includes:

[0026] Acquire a plurality of continuous color image frames within a preset time length related to the color image;

[0027] Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and obtaining a cub identification result corresponding to each frame of color image output by the target detection model;

[0028] If it is determined that there is no cub in the buttocks area of ​​the target livestock according to each of the cub identification results, then the livestock's delivery behavior is determined to be a dystocia.

[0029] A method for monitoring livestock delivery behavior provided by the present invention further comprises:

[0030] Generate corresponding status instructions according to different livestock birthing behavior judgment results, and use the status instructions to drive the programmable signal light to issue corresponding status prompts;

[0031] The status prompt includes at least one of a light prompt, a sound prompt and a light flashing frequency prompt.

[0032] A method for monitoring livestock delivery behavior provided by the present invention further comprises:

[0033] The continuous multiple frames of color images collected within the preset time, the thermal infrared images corresponding to each frame of the color image, and the results of livestock delivery behavior judgment within the preset time are stored for the pig farm manager to call and display on the touch screen.

[0034] In a second aspect, the present invention further provides a livestock delivery behavior monitoring device, comprising:

[0035] An image acquisition unit, used for acquiring a thermal infrared image and a color image including a hip area of ​​a target livestock;

[0036] A model recognition unit, used for inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image;

[0037] A cub judgment unit, when it is determined according to the cub recognition result that there is a cub in the buttocks area of ​​the target livestock, obtains the first area coordinates of the cub target frame corresponding to the cub in the color image;

[0038] An image matching unit, used for converting the first area coordinates into second area coordinates on the thermal infrared image to determine a cub target frame area of ​​the cub in the thermal infrared image;

[0039] The result determination unit is used to determine the result of livestock delivery behavior according to the temperature distribution in the young target frame area.

[0040] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for monitoring livestock delivery behavior are implemented.

[0041] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the livestock parturition behavior monitoring methods described above.

[0042] The livestock delivery behavior monitoring method and device provided by the present invention adopt the multi-data fusion technology of color images and thermal infrared images, monitor the livestock delivery behavior by fusing temperature and target movement detection, have high recognition accuracy, and effectively improve the degree of automation of livestock breeding. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is one of the flow charts of the livestock parturition behavior monitoring method provided by the present invention;

[0045] Figure 2 This is one of the structural schematic diagrams of the livestock delivery behavior monitoring device provided by the present invention;

[0046] Figure 3 It is a schematic diagram of the network structure of a YOLOv5s network model provided by the present invention;

[0047] Figure 4 It is a flow chart of the Mosaic data enhancement method provided by the present invention;

[0048] Figure 5 is a schematic diagram of the Focus structure slicing operation provided by the present invention;

[0049] Figure 6 This is the second flow chart of the livestock parturition behavior monitoring method provided by the present invention;

[0050] Figure 7This is the second structural diagram of the livestock delivery behavior monitoring device provided by the present invention.

[0051] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0055] Combine the following Figure 1-Figure 8 The present invention describes a method and device for monitoring livestock delivery behavior provided by an embodiment of the present invention.

[0056] Figure 1 FIG. 1 is one of the flow charts of the method for monitoring livestock parturition behavior provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0057] Step 101: Acquire a thermal infrared image and a color image containing the hip area of ​​the target livestock;

[0058] Step 102: inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image;

[0059] Step 103: if it is determined that there is a cub in the buttocks area of ​​the target livestock according to the cub identification result, the first area coordinates of the cub target frame corresponding to the cub in the color image are obtained;

[0060] Step 104: converting the first area coordinates into second area coordinates on the thermal infrared image to determine a cub target frame area of ​​the cub in the thermal infrared image;

[0061] Step 105: Determine the livestock delivery behavior judgment result according to the temperature distribution in the young target frame area.

[0062] It should be noted in advance that the livestock delivery behavior monitoring device provided by the present invention can be used to monitor the delivery behavior of livestock in cow sheds, pig sheds, sheep sheds, etc. In order to more clearly and concisely illustrate the specific scheme of the present invention, in the subsequent embodiments, sows giving birth to piglets are used as an example for explanation, which is not regarded as a specific limitation on the scope of protection of the present invention.

[0063] Figure 2 FIG. 1 is one of the structural schematic diagrams of the livestock delivery behavior monitoring device provided by the present invention. Figure 2 As shown, the livestock delivery behavior monitoring device integrates edge computing technology, mainly including but not limited to edge computing units, thermal infrared sensors, full-color cameras, signal lights, touch screens, power modules, etc.

[0064] Optionally, the edge computing unit has built-in multiple USB interfaces, multimedia interfaces (High Definition Multimedia Interface, HDMI), Wi-Fi communication modules, and large-capacity memory cards (such as 64G memory cards), which are mainly used for reading and processing thermal infrared images and color images.

[0065] Among them, the thermal infrared sensor can be a high-resolution sensor with an infrared temperature measurement resolution of 0.25°C and a temperature measurement range of -20°C to 80°C. It is mainly used to obtain thermal infrared images to monitor the temperature of the sow's buttocks area and the body temperature of the piglets.

[0066] The full-color camera can use a full-color autofocus USB interface camera to meet the needs of high-definition color image acquisition at different distances and in all weather conditions.

[0067] The display screen can be a touch screen, which can realize functions such as adjustment of preset parameters, display of current data, and playback of historical data.

[0068] As an optional solution, in step 101, a livestock delivery behavior monitoring device can be installed behind the sow's farrowing bed, and the thermal infrared sensor and the full-color camera can be adjusted to at least be able to capture the sow's buttocks area (preferably facing the sow's buttocks area) to collect thermal infrared images and color images containing the sow's buttocks area around the clock.

[0069] As another optional embodiment, the livestock birthing behavior monitoring device can be set up on a movable platform, which can be a kind of electrically driven movable chassis. In addition, tracks can also be installed on the top of the livestock house in advance, and the movable platform can run on this track through an electric pulley.

[0070] It should be noted that the present invention can automatically patrol the livestock house according to actual monitoring needs by installing the livestock delivery behavior monitoring device on a movable platform.

[0071] Optionally, the movable platform can receive remote control commands to run to any preset target position according to the control commands. For example, a target position is set on the top of each livestock house in advance. During the inspection process of the movable platform, after arriving at a target position each time, it stays for a preset time, and performs the livestock delivery behavior monitoring described in the above embodiment within this preset time. If it is determined that the livestock delivery behavior judgment result is that the delivery behavior has not occurred, the next target position will continue to be monitored, and the cyclic monitoring of all target positions will be completed in a cycle; if it is determined that the livestock delivery behavior judgment result is delivery or other delivery status, the livestock delivery behavior monitoring will continue at a target position until the delivery is completed, and then it will automatically run to the next target position.

[0072] As another optional embodiment, the movable platform may also receive control instructions sent remotely by a user and directly move to any target location.

[0073] In the present invention, the thermal infrared sensor and the full-color camera work in coordination, and when the full-color camera takes a color image frame, the thermal infrared camera is driven to take a thermal infrared image frame. Alternatively, a thermal infrared image and a color image are taken at this time, and the thermal infrared image and the color image taken at each time point are matched one by one according to the shooting time, so as to provide a basis for the multi-data fusion of the color image and the thermal infrared image in the later stage.

[0074] In step 102, the color images collected in the continuous time period are sequentially input into the pre-trained target detection model to obtain the cub recognition result output by the target detection model.

[0075] There are four main target detection models in the existing YOLOv5 algorithm, namely YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x network models. Among them, the YOLOv5s network has the smallest depth and the smallest feature map width. The other three are continuously deepened and widened on this basis, that is, the YOLOv5s network model is the most lightweight.

[0076] In view of this, the target detection model built into the livestock delivery behavior monitoring device provided by the present invention can use the YOLOv5s network model to integrate the various processes of piglet target detection into a neural network model. By resizing each frame of color image and inputting it into a pre-trained YOLOv5s network model, it can preliminarily determine whether there is image information related to newly born piglets in each frame of color image. If so, the piglet target frame where the piglet is located can be accurately predicted.

[0077] Figure 3 It is a schematic diagram of the network structure of a YOLOv5s network model provided by the present invention, such as Figure 3 As shown in the figure, it is mainly composed of four parts: input end, backbone network Backbone, Neck, and Predicition.

[0078] Optionally, its input end mainly includes the following functional modules: Mosaic data enhancement, adaptive anchor box calculation, adaptive image scaling operation module; Backbone mainly includes Focus structure and CSP structure; Neck mainly includes FPN structure and PAN structure; Prediction mainly includes border optimization loss function (GIOU_Loss loss function).

[0079] Figure 4 It is a flow chart of the Mosaic data enhancement method provided by the present invention, such as Figure 4As shown in the figure, the Mosaic data enhancement method is mainly: every 4 frames of input color images can be spliced ​​by random scaling, random cropping, and random arrangement to generate a frame of image to be detected, which can effectively enrich the background of the target to be detected and some small targets, and when performing batch normalization, the data of four color pictures can be calculated at one time, so that the size of the mini-batch does not need to be very large, and a GPU can achieve better results, so it can effectively improve the efficiency of model recognition and meet the requirements of lightweight.

[0080] In addition, in the object detection model provided by the present invention, the adaptive anchor box calculation automatically learns the size of the anchor points according to the new dataset labels. The custom dataset is analyzed using the k-means and genetic learning algorithms to obtain a preset anchor box suitable for the prediction of the object bounding box in the custom dataset.

[0081] The above involves adaptive image operations, which are mainly based on the fact that in commonly used target detection algorithms, different images have different lengths and widths. Therefore, a common method is to uniformly scale the original image to a standard size and then send it to the detection network. This does not change the aspect ratio of the original image, and other missing areas are filled in with black. This image preprocessing method can reduce the amount of calculation during model training and increase the target detection speed.

[0082] The Focus module mentioned above refers to the slicing operation of the color image before it enters the backbone in the target detection model.

[0083] Figure 5 is a schematic diagram of the Focus structure slicing operation provided by the present invention, and the specific operations may include:

[0084] A value is extracted from every other pixel in a frame of color image, which is similar to neighboring downsampling. In this way, four images to be analyzed can be determined for each frame of color image. The four images to be analyzed are complementary and have similar image features, but there is no loss of image information.

[0085] like Figure 5 As shown, the four pixel coordinates (0, 0), (0, 1), (1, 0), and (1, 1) at the upper left corner of a frame of color image are sampled at alternate points and then spliced, so that each frame of color image can be downsampled into four color images.

[0086] In this way, the width and height information of the image to be analyzed are concentrated in the channel space, which is equivalent to expanding the input channel by 4 times. That is, the new image spliced ​​together from the four images to be analyzed becomes a 12-channel mode compared to the original three-channel mode of the color image.

[0087] Finally, the new color image is subjected to a convolution operation, and finally a double-downsampled feature map is obtained without information loss.

[0088] Taking YOLOv5s as an example, the original 640*640*3 image is input into the Focus structure, and a slicing operation is used to first convert it into a 320*320*12 feature map, and then a convolution operation is performed to finally convert it into a 320*320*32 feature map.

[0089] Furthermore, the CSP structures mentioned above include: CSP1_X structure and CSP2_X structure.

[0090] Two CSP structures are designed in YOLOV5s. Taking the YOLOv5s network as an example, the CSP1_X structure is applied to the Backbone network, and the other CSP2_X structure is applied to the Neck. The CSP structure integrates the gradient changes from beginning to end into the feature map, which reduces the amount of calculation while ensuring accuracy.

[0091] In the target detection model Yolov5s adopted in the present invention, the border optimization loss GIOU_Loss is used as the loss function of the Bounding box.

[0092] The expression of GIOU_Loss loss function is:

[0093]

[0094] Among them, Ac is the area of ​​the smallest circumscribed rectangle of the predicted box and the true box; U is the union of the predicted box and the true box; IoU is the intersection over union of the predicted box and the true box.

[0095] As an explanation, the actual operation principle of the above GIOU_Loss loss function includes: first calculate the minimum enclosed area Ac of the predicted box and the real box (including the area of ​​the minimum circumscribed rectangular box of the predicted box and the real box); then calculate the IoU, and then calculate the proportion of the area in the closure area that does not belong to the two boxes; finally, subtract this proportion from the IoU to get the GIoU.

[0096] By inputting each frame of color image into the above-mentioned YOLOv5 target detection model, the cub recognition result corresponding to each frame of color image output by it is obtained.

[0097] In step 103, if it is determined according to the piglet recognition result that there is a piglet in a certain frame of color image, the target detection model can determine the piglet target frame corresponding to the piglet in the color image.

[0098] During the actual delivery process of the sow, the body temperature of the newly born piglets is 1 to 2 degrees Celsius higher than that of other piglets. Based on this principle, in the livestock delivery behavior monitoring method provided by the present invention, after the piglet target frame in each frame of the color image is obtained by using the target detection model, its coordinates in the color image are recorded as the first area coordinates.

[0099] Then, the coordinate conversion relationship between the thermal infrared sensor 1 and the camera 2 is obtained, and then according to this coordinate conversion relationship, the first area coordinates are first converted into the second area coordinates on the thermal infrared image, so as to realize the operations such as scaling and translation transformation of the piglet target frame in the color image, and map it to the corresponding piglet target frame area in the thermal infrared image.

[0100] Then, the cub target frame area in the thermal infrared image is mapped to the thermal infrared data matrix, and the temperature distribution in the cub target frame area can be obtained.

[0101] The step of obtaining the coordinate conversion relationship between the thermal infrared sensor 1 and the camera 2 may include:

[0102] Before the coordinate conversion process, a point heat source is photographed simultaneously with a camera and a thermal infrared camera, and the coordinates of the point heat source in the color image and its coordinates in the thermal infrared image are recorded. After multiple measurements, a linear regression method can be used to obtain the conversion equation between the two coordinates.

[0103] In step 105, after obtaining the temperature distribution in the piglet target frame area according to the thermal infrared image, the livestock delivery behavior judgment result can be determined based on the information provided by the temperature distribution. This embodiment does not make specific limitations on this. For example, whether delivery occurs can be determined based on whether the highest temperature value in the piglet target frame area is greater than the minimum temperature requirement when the piglet just gives birth (i.e., the temperature threshold for piglets just given birth).

[0104] The livestock delivery behavior monitoring method provided by the present invention adopts the multi-data fusion technology of color images and thermal infrared images, monitors the livestock delivery behavior by fusing temperature and target movement detection, has high recognition accuracy, and effectively improves the degree of automation of livestock breeding.

[0105] Based on the content of the above embodiment, as an optional embodiment, determining the livestock delivery behavior judgment result according to the temperature distribution in the young target frame area includes:

[0106] If the highest temperature in the cub target frame area is greater than the temperature threshold of the cub just born, the livestock birthing behavior judgment result is determined to be birthing, otherwise the livestock birthing behavior judgment result is determined to be other birthing states.

[0107] Generally speaking, the sow's delivery behavior status can be divided into: stillbirth, delivery (delivery can also be subdivided into pre-delivery, mid-delivery, late-delivery and completed delivery), dystocia and other states.

[0108] Figure 6 This is the second flow chart of the livestock delivery behavior monitoring method provided by the present invention, combined with Figure 6 As shown, after any frame of color image of the target livestock hip area is input into the YOLOv5s target detection model, the cub target frame area in the thermal infrared image corresponding to the color image is obtained according to the method of the above embodiment, and the temperature distribution in the cub target frame area is obtained.

[0109] In this embodiment, the maximum temperature in the young target frame area is compared with the temperature threshold of the newly delivered young cubs. If the maximum temperature in the young target frame area is 36 degrees Celsius, and the temperature threshold of the newly delivered young cubs is 34 degrees Celsius, then it can be determined that the livestock delivery behavior judgment result is delivery; if the maximum temperature in the young target frame area is only 33 degrees Celsius, which is less than the temperature threshold of the newly delivered young cubs, then it means that the livestock delivery behavior result is not in the delivery state, such as: in a dead fetus or delivery has been completed for a period of time (collectively referred to as other delivery states).

[0110] As another optional embodiment, the above-mentioned determination of the livestock delivery behavior judgment result based on the temperature distribution in the cub target frame area may also include:

[0111] Calculate the median temperature in the cub target frame area as the average temperature of the cub target frame area;

[0112] Determine a surrounding area within a preset range outside the cub target frame area, and calculate the median temperature of the surrounding area as the average temperature of the surrounding area;

[0113] If the difference between the median temperature of the pup target frame area (equivalent to the average temperature of the median temperature of the pup target frame area) and the median temperature of the surrounding area (equivalent to the average temperature of the surrounding area) is greater than a preset temperature difference threshold, the livestock delivery behavior judgment result is determined to be delivery; otherwise, the livestock delivery behavior judgment result is determined to be other delivery states.

[0114] When the sow gives birth, the temperature at the location of the piglets will be higher than the temperature of the surrounding area. Therefore, in this embodiment, the median temperature of the piglet target frame area is compared with the median temperature of the surrounding area to determine the result of livestock birthing behavior.

[0115] It should be noted that the cub target frame area is determined by mapping the cub target frame identified by the target detection model to the thermal infrared image. The size of the surrounding area corresponding to the cub target frame area can be determined according to the type of livestock. Generally speaking, the larger the size of the livestock, the larger the surrounding area.

[0116] Subtract the average temperature of the pup target frame area from the average temperature of the surrounding area. Assuming that the average temperature of the pup target frame area is 38 degrees Celsius and the average temperature of the surrounding area is 35 degrees Celsius, the difference between the two is 3 degrees Celsius, which is greater than the preset temperature difference threshold (assuming 1.5 degrees Celsius), indicating that the livestock delivery behavior judgment result is delivery. Correspondingly, if the difference between the two is less than or equal to 1.5 degrees Celsius, it means that the livestock delivery behavior judgment result is other delivery states.

[0117] The livestock delivery behavior monitoring method provided by the present invention determines the average temperature of the area where the piglets are located through the multi-data fusion technology of color images and thermal infrared images, and compares the temperature with the surrounding area. It can accurately judge whether the sow is in the delivery state with high detection accuracy.

[0118] Based on the content of the above embodiment, as an optional embodiment, after determining that the livestock delivery behavior judgment result is other delivery states, it also includes:

[0119] Acquire a plurality of continuous color image frames within a preset time length related to the color image;

[0120] Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and determining the cub target frame corresponding to each frame of color image according to the cub recognition result corresponding to each frame of color image output by the target detection model;

[0121] Using the target tracking model to track all piglet target frames, and obtaining the activity trajectory of the piglets within the preset time period;

[0122] If it is determined according to the activity trajectory that the cub continues to move within the preset time period, then the other delivery state is determined to be completion of delivery;

[0123] If it is determined according to the activity trajectory that the cub is stationary within the preset time period, and the highest temperature in the cub target frame area is less than the dead fetus temperature threshold, then the other delivery state is determined to be a dead fetus.

[0124] Optionally, the present invention also pre-loads a target tracking model, such as using an IoU multi-frame association algorithm, and analyzes the pixel coordinates of the piglet target frame in the image obtained in real time by the target detection model, and compares the intersection and union ratio of the pixel coordinates of the piglet target frame of the previous frame with the pixel coordinates of the target frame of the current frame. If the IoU of multiple consecutive frames is greater than a preset value, it can be determined that the content of the target frame is the same piglet, thereby achieving accurate tracking of the target.

[0125] Among them, the IoU algorithm calculation formula is as follows:

[0126]

[0127] Among them: C is the piglet target frame of the current frame; G is the piglet target frame of the previous frame.

[0128] Combination Figure 6 As shown, when it is determined that the sow is in other delivery states based on the obtained temperature distribution of the piglet target frame area, it is necessary to further determine whether the sow is in a state of stillbirth, dystocia or other states.

[0129] First, multiple frames of color images collected within a preset time (such as 2 hours) need to be input into a pre-trained target detection model in sequence to identify the target frame of the cub in each frame of the color image.

[0130] Then, the movement of the piglets is predicted by analyzing the activity trajectory of the piglet target frame in multiple consecutive color images.

[0131] If it is determined that the piglets have been in motion based on their activity trajectories within a preset period of time, it means that the piglets have been eating freely in the delivery bed, so it can be determined that the result of the livestock delivery behavior is that the delivery is complete.

[0132] As another possibility, if the distance of the piglet's activity trajectory within the preset time is almost zero, it means that the delivered piglet is almost in a static state within the preset time; combined with the method adopted in the above embodiment, it is determined that the highest temperature (or the median temperature) in the piglet target frame area is less than the stillbirth temperature threshold, then it can be concluded that the delivered piglet is in a stillbirth state.

[0133] Among them, the temperature threshold for stillbirth refers to the lowest temperature at which the piglets are alive. If the temperature is lower than this, it can be basically determined that the piglets are stillborn, for example:

[0134] Assuming that the temperature threshold of stillbirth is 28 degrees Celsius, color images and thermal infrared images are acquired in real time within 2 hours.

[0135] The edge computing module obtains the coordinate position of the piglet target frame in the image, coordinates the first area coordinates in the color image, and converts them into the second area coordinates of the corresponding thermal infrared image; then it maps it to the thermal infrared temperature matrix to calculate the highest temperature in the piglet target frame area.

[0136] If the maximum temperature is 36 degrees Celsius, which is higher than the temperature threshold of 34 degrees Celsius for newly born piglets, it can be determined that the sow is in the process of giving birth.

[0137] On the contrary, if the maximum temperature is 26 degrees Celsius, which is lower than the temperature threshold of 34 degrees Celsius for newly born pups, it means that the result of livestock delivery behavior is other delivery status.

[0138] At this time, each frame of color image within 2 consecutive hours (assuming that the sampling frequency is to collect one frame of color image every 5 minutes) is sequentially input into the target detection model, and the target frame of the cub in each frame of color image can be determined;

[0139] By using the target tracking model to track all piglet target frames, the activity trajectory of the piglets in these two hours can be obtained.

[0140] If it is determined based on the activity trajectory that the piglet remains basically still within these two hours and the maximum temperature is lower than the stillbirth temperature threshold of 28 degrees Celsius, it can be determined that the piglet is stillborn.

[0141] If it is determined based on the activity trajectory that the piglets have been in motion during these two hours, it can be determined that the piglets are eating freely in the farrowing crate.

[0142] Based on the content of the above embodiment, as another optional embodiment, if it is determined that there is no young animal in the buttocks area of ​​the target livestock according to the young animal identification result, the method further includes:

[0143] Acquire a plurality of continuous color image frames within a preset time length related to the color image;

[0144] Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and obtaining a cub identification result corresponding to each frame of color image output by the target detection model;

[0145] If it is determined that there is no cub in the buttocks area of ​​the target livestock according to each of the cub identification results, then the livestock's delivery behavior is determined to be a dystocia.

[0146] Combination Figure 6 As shown, if in the above steps, the target detection model is used to identify the continuous multiple frames of color images collected within 2 hours, but according to the piglet identification results corresponding to each frame of color image, it is determined that there is no piglet in the buttocks area of ​​the target livestock, it means that the sow has a dystocia.

[0147] The livestock delivery behavior monitoring method provided by the present invention combines a target detection model with a target tracking model to simulate the activity trajectory of the target frame where the identified cub is located, and then integrates the temperature distribution in the target frame where the cub is located to comprehensively judge the livestock delivery status, thereby greatly improving the degree of automation and the accuracy of recognition.

[0148] Based on the contents of the above embodiments, as an optional embodiment, the livestock delivery behavior monitoring method provided by the present invention further includes: generating corresponding status instructions according to different livestock delivery behavior judgment results, and using the status instructions to drive the programmable signal light to issue a corresponding status prompt; the status prompt includes at least one of a light prompt, a sound prompt and a light flashing frequency prompt.

[0149] by Figure 6 Taking the example shown, since the results of livestock delivery behavior judgment mainly include stillbirth, delivery, dystocia, etc., it is pre-set that when the result of livestock delivery behavior judgment is stillbirth, the status prompt is a flashing red signal light; when the result of livestock delivery behavior judgment is delivery, the status prompt is a flashing green signal light; when the result of livestock delivery behavior judgment is dystocia, the status prompt is a flashing yellow signal light.

[0150] It should be noted that the above-mentioned programmable signal light gives different status prompts according to the results of the livestock's parturition behavior judgment, and can be changed and set according to the actual programming of the programmable signal light.

[0151] For example, when it is determined that the birthing behavior of livestock is dystocia, the yellow signal light of the programmable signal light can be controlled to flash, and a sound prompt can be issued at the same time. The sound prompt can be a voice broadcast or a pre-recorded alarm sound.

[0152] Furthermore, the frequency of flashing of the yellow signal light may be increased, and the loudness or tone of the sound prompt may be increased according to the duration of the dystocia.

[0153] In addition, a manual control switch can be added to the programmable signal light to allow the farm manager to control its operating status. For example, when the farm manager intervenes, the flashing alarm of the signal light can be removed through the manual control switch.

[0154] In addition, the livestock delivery behavior judgment results and related delivery information can be sent to off-site managers through the Wi-Fi communication module using wireless communication.

[0155] The livestock delivery behavior monitoring method provided by the present invention sets a signal light to judge the result according to the identified livestock delivery behavior in real time, and displays it in the form of light and sound, so that the pig farm manager can intuitively obtain the livestock delivery status and make a quick response to the corresponding situation.

[0156] Based on the contents of the above embodiments, as an optional embodiment, the livestock delivery behavior monitoring method provided by the present invention further includes storing the continuous multiple frames of color images collected within the preset time period, the thermal infrared images corresponding to each frame of the color image, and the livestock delivery behavior judgment results within the preset time period, so that they can be called up by the pig farm manager and displayed on the touch display screen.

[0157] The touch display screen is used to receive input information from the pig farm manager; the input information includes at least one of the preset parameters of the target detection model and the target tracking model, the setting parameters of the programmable signal light, the delivery bed number information, and the ear tag information.

[0158] There are differences in the sow breeding environment of different pig farms, and the recognition rate of sow farrowing monitoring devices will vary to a certain extent. The recognition accuracy can be improved by adjusting the sensitivity of the target detection model and the target tracking model.

[0159] Specifically, when the display screen is a touch screen, human-computer interaction is performed through the multi-point touch screen, which can be mainly used to realize the adjustment of preset parameters (such as parameters related to the target detection model and the target tracking model), the setting of various thresholds (such as the temperature threshold for piglets that have just been born, the preset dystocia time interval threshold, etc.), current data display, historical data playback, etc.

[0160] In addition, the farm manager inputs the farrowing bed number information through the touch screen and combines it with the pre-recorded sow ear tag information to facilitate the management and record keeping of the sow's farrowing process.

[0161] The livestock delivery behavior monitoring method provided by the present invention can meet the interactive needs of the pig farm manager for the monitoring device by setting a display screen, especially a touch display screen, such as manually setting and adjusting preset parameters, retrieving historical monitoring data to realize big data analysis, or performing health status analysis on any sow, etc., thereby ensuring the flexibility of the monitoring device and improving the use preference of the pig farm manager.

[0162] Figure 7 FIG. 2 is a second structural diagram of the livestock delivery behavior monitoring device provided by the present invention. Figure 7 As shown, it mainly includes but is not limited to an image acquisition unit 1, a model recognition unit 2, a cub judgment unit 3, an image matching unit 4 and a result judgment unit 5, wherein:

[0163] The image acquisition unit 1 is mainly used to obtain a thermal infrared image and a color image containing the target livestock's hip area;

[0164] The model recognition unit 2 is mainly used to input the color image into a pre-trained target detection model to obtain a cub recognition result of the color image;

[0165] The cub judgment unit 3 mainly obtains the first area coordinates of the cub target frame corresponding to the cub in the color image when it is determined that there is a cub in the buttocks area of ​​the target livestock according to the cub recognition result;

[0166] The image matching unit 4 is mainly used to convert the first area coordinates into second area coordinates on the thermal infrared image to determine the cub target frame area of ​​the cub in the thermal infrared image;

[0167] The result determination unit 5 is mainly used to determine the result of livestock delivery behavior according to the temperature distribution in the young target frame area.

[0168] Optionally, the livestock delivery behavior monitoring device provided by the present invention first recognizes the color image collected by the image acquisition unit 1 by calling the target detection model in the model recognition unit 2 to obtain the piglet recognition result. When the piglet judgment unit 3 determines that there is a piglet in the buttocks area of ​​the target livestock according to the piglet recognition result, it obtains the coordinate position of the piglet target frame in the color image.

[0169] The image matching unit 4 performs coordinate transformation on the piglet target frame to convert it into the coordinate position of the corresponding thermal infrared image, and maps it into the thermal infrared temperature matrix.

[0170] After the result determination unit 5 calculates the temperature of the piglet target frame area, or obtains the highest temperature in the piglet target frame area, if it is determined that the temperature is greater than the temperature threshold of the piglets that have just given birth, it can be determined that the sow has given birth.

[0171] On the contrary, if the temperature threshold of piglets just farrowed is not reached, the body temperature and activity trajectory of the piglets are further determined. The temperature threshold of piglets just farrowed is the lowest body temperature of the piglets.

[0172] If the temperature of the monitored piglet drops to the preset dead fetus temperature threshold, and the target tracking model determines the center point of the piglet target frame detected by the target detection model, and the distance of its activity trajectory is almost 0 within the preset time, it can be judged as a dead fetus. Correspondingly, if the temperature of the monitored piglet is within the preset value range of the live piglet temperature, and the distance of its activity trajectory is greater than 0 within the preset time, it is judged that the piglet is feeding freely in the delivery bed.

[0173] The livestock delivery behavior monitoring device provided by the present invention adopts the multi-data fusion technology of color images and thermal infrared images, monitors the livestock delivery behavior by fusing temperature and target movement detection, has high recognition accuracy, and effectively improves the degree of automation of livestock breeding.

[0174] It should be noted that the livestock delivery behavior monitoring device provided in the embodiment of the present invention can execute the livestock delivery behavior monitoring method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0175] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the livestock delivery behavior monitoring method, which includes: obtaining a thermal infrared image and a color image containing the target livestock hip area; inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image; if it is determined according to the cub recognition result that there is a cub in the target livestock hip area, then obtaining the first area coordinates of the cub target frame corresponding to the cub in the color image; converting the first area coordinates into the second area coordinates on the thermal infrared image to determine the cub target frame area of ​​the cub in the thermal infrared image; and determining the livestock delivery behavior judgment result according to the temperature distribution in the cub target frame area.

[0176] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0177] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the livestock delivery behavior monitoring method provided by the above methods, and the method includes: obtaining a thermal infrared image and a color image containing the target livestock's hip area; inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image; if it is determined that there is a cub in the target livestock's hip area based on the cub recognition result, then obtaining the first area coordinates of the cub target box corresponding to the cub in the color image; converting the first area coordinates into second area coordinates on the thermal infrared image to determine the cub target box area of ​​the cub in the thermal infrared image; and determining the livestock delivery behavior judgment result based on the temperature distribution in the cub target box area.

[0178] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the livestock delivery behavior monitoring method provided in the above-mentioned embodiments, the method comprising: obtaining a thermal infrared image and a color image including the hip area of ​​the target livestock; inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image; if it is determined based on the cub recognition result that there is a cub in the hip area of ​​the target livestock, then obtaining the first area coordinates of the cub target frame corresponding to the cub in the color image; converting the first area coordinates into second area coordinates on the thermal infrared image to determine the cub target frame area of ​​the cub in the thermal infrared image; and determining the livestock delivery behavior judgment result based on the temperature distribution in the cub target frame area.

[0179] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring livestock parturition behavior, characterized in that: include: Acquire a thermal infrared image and a color image containing the target livestock's hip area, wherein the thermal infrared image and the color image are in one-to-one correspondence according to the shooting time point; Inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image; If it is determined according to the cub identification result that there is a cub in the buttocks area of ​​the target livestock, obtaining the first area coordinates of the cub target frame corresponding to the cub in the color image; Converting the first area coordinates into second area coordinates on the thermal infrared image to determine a cub target frame area of ​​the cub in the thermal infrared image; According to the temperature distribution in the target frame area of ​​the young animal, the livestock delivery behavior judgment result is determined, including: Calculate the median temperature in the cub target frame area as the average temperature of the cub target frame area; Determine a surrounding area within a preset range outside the cub target frame area, and calculate the median temperature of the surrounding area as the average temperature of the surrounding area; If the difference between the average temperature of the target area of ​​the young pig and the average temperature of the surrounding area is greater than a preset temperature difference threshold, the livestock delivery behavior judgment result is determined to be delivery; otherwise, the livestock delivery behavior judgment result is determined to be other delivery states.

2. The method for monitoring livestock delivery behavior according to claim 1, characterized in that: Determining the livestock delivery behavior judgment result according to the temperature distribution in the young target frame area includes: If the highest temperature in the cub target frame area is greater than the temperature threshold of the cub just born, the livestock birthing behavior judgment result is determined to be birthing, otherwise the livestock birthing behavior judgment result is determined to be other birthing states.

3. The method for monitoring livestock delivery behavior according to any one of claims 1 or 2, characterized in that: After determining that the livestock delivery behavior is determined to be other delivery states, the method further includes: Acquire a plurality of continuous color image frames within a preset time length related to the color image; Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and determining the cub target frame corresponding to each frame of color image according to the cub recognition result corresponding to each frame of color image output by the target detection model; Using the target tracking model to track all piglet target frames, and obtaining the activity trajectory of the piglets within the preset time period; If it is determined according to the activity trajectory that the cub continues to move within the preset time period, then the other delivery state is determined to be completion of delivery; If it is determined according to the activity trajectory that the cub is stationary within the preset time period, and the highest temperature in the cub target frame area is less than the dead fetus temperature threshold, then the other delivery state is determined to be a dead fetus.

4. The method for monitoring livestock delivery behavior according to claim 1, characterized in that: If it is determined according to the result of the cub identification that there is no cub in the buttocks area of ​​the target livestock, the method further includes: Acquire a plurality of continuous color image frames within a preset time length related to the color image; Inputting the continuous multiple frames of color images into the trained target detection model in sequence, and obtaining a cub identification result corresponding to each frame of color image output by the target detection model; If it is determined that there is no cub in the buttocks area of ​​the target livestock according to each of the cub identification results, then the livestock's delivery behavior is determined to be a dystocia.

5. The method for monitoring livestock delivery behavior according to claim 1, characterized in that: Also includes: Generate corresponding status instructions according to different livestock birthing behavior judgment results, and use the status instructions to drive the programmable signal light to issue corresponding status prompts; The status prompt includes at least one of a light prompt, a sound prompt and a light flashing frequency prompt.

6. The method for monitoring livestock delivery behavior according to claim 3, characterized in that: Also includes: The continuous multiple frames of color images collected within the preset time, the thermal infrared images corresponding to each frame of the color image, and the results of livestock delivery behavior judgment within the preset time are stored for the pig farm manager to call and display on the touch screen.

7. A livestock birthing behavior monitoring device, characterized in that: include: An image acquisition unit, used to acquire a thermal infrared image and a color image containing a target livestock hip area, wherein the thermal infrared image and the color image are in one-to-one correspondence according to a shooting time point; A model recognition unit, used for inputting the color image into a pre-trained target detection model to obtain a cub recognition result of the color image; A cub judgment unit, when it is determined according to the cub recognition result that there is a cub in the buttocks area of ​​the target livestock, obtains the first area coordinates of the cub target frame corresponding to the cub in the color image; An image matching unit, used for converting the first area coordinates into second area coordinates on the thermal infrared image to determine a cub target frame area of ​​the cub in the thermal infrared image; The result determination unit is used to determine the result of livestock delivery behavior according to the temperature distribution in the target frame area of ​​the young piglets, including: Calculate the median temperature in the cub target frame area as the average temperature of the cub target frame area; Determine a surrounding area within a preset range outside the cub target frame area, and calculate the median temperature of the surrounding area as the average temperature of the surrounding area; If the difference between the average temperature of the target area of ​​the young pig and the average temperature of the surrounding area is greater than a preset temperature difference threshold, the livestock delivery behavior judgment result is determined to be delivery; otherwise, the livestock delivery behavior judgment result is determined to be other delivery states.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the livestock parturition behavior monitoring method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the livestock parturition behavior monitoring method according to any one of claims 1 to 6 are implemented.

Images