System and method for counting pigs in a pig population based on video

Abstract

A method and system for counting pigs in a hog house are provided. The system includes a computing device and an imaging device. The computing device has a processor and a storage device having computer executable code stored therein. The computer executable code is configured to: receive images captured from one end of the house to the other end of the house; detect key points in the images using a neural network; identify pig skeletons by associating a plurality of related key points; track the skeletons in the images to obtain trajectories; divide each image into active zones and deactive zones; assign a spatial value of 0 to the skeletons in the active zones and a spatial value of 1 to the skeletons in the deactive zones; sum a first order difference of the spatial values for each trajectory to obtain a trajectory count; and add the trajectory counts to obtain a pig count.

Description

[0001] Cross-references

[0002] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citations and / or discussions of such references are provided solely to clarify the description of this disclosure and do not imply that any such reference is prior art to this disclosure. All references cited or discussed in this specification are incorporated herein by reference in their entirety, to the same extent as each individual reference is incorporated by reference individually. Technical Field

[0003] This disclosure generally relates to counting large numbers of animals, and more specifically, to systems and methods for counting a group of pigs in a herd based on video. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. Within the scope of this background description, the present inventor's work, and descriptions that may not conform to prior art at the time of filing, are neither explicitly nor implicitly acknowledged as prior art to this disclosure.

[0005] Pig pens typically house a large number of pigs, and frequent headcount is a critical management task for large-scale pig farms. On one hand, pigs are usually moved to different pens at different growth stages or grouped by size into separate large enclosures, and farmers need to know how many pigs are in each enclosure. On the other hand, comparing the count with the actual number of pigs allows for early detection of unexpected events, such as lost pigs. However, walking around the pigpens to count large numbers of pigs is labor-intensive.

[0006] Therefore, there is an unresolved need in this field to address the aforementioned defects and shortcomings. Summary of the Invention

[0007] In some aspects, this disclosure relates to a system for counting animals in a dwelling. The system includes a computing device and an imaging device. The computing device includes a processor and a storage device storing computer-executable code. The computer-executable code, when executed at the processor, is configured to:

[0008] Receive multiple images captured by the imaging device from one end of the building to the opposite end of the building;

[0009] Use a neural network to detect key points in each image;

[0010] The skeleton of an animal is identified by associating multiple key points in the image, where each skeleton corresponds to one animal;

[0011] Track the skeleton identified in the image to obtain the trajectory of the skeleton;

[0012] Each image is divided into an active region and a deactivated region;

[0013] Assign a first spatial value to the skeleton in the activated region, and assign a second spatial value to the skeleton in the deactivated region;

[0014] A trajectory count is provided for each trajectory by summing the first spatial value and the second spatial value of a corresponding skeleton in a given trajectory; and

[0015] The animal count is obtained by summing the track counts.

[0016] In some embodiments, each skeleton includes at least one of five types of keypoints: a head keypoint corresponding to the animal's head, a neck keypoint corresponding to the animal's neck, a back keypoint corresponding to the animal's back, a waist keypoint corresponding to the animal's waist, and a tail keypoint corresponding to the animal's tail. Each detected keypoint includes a classification and a vector, wherein the classification indicates which of the five keypoint types the detected keypoint belongs to, and the vector indicates a direction toward at least one detected keypoint belonging to the same skeleton as the detected keypoint. In some embodiments, the back keypoint is defined as the root keypoint, the neck keypoint and the waist keypoint are labeled as the same type, and the head keypoint and the tail keypoint are labeled as the same type.

[0017] In some embodiments, the first space value is 0 and the second space value is 1.

[0018] In some embodiments, the size of the active region and the size of the deactivated region are substantially equal, and the active region is located in front of the deactivated region in the direction of movement of the imaging device.

[0019] In some embodiments, the computer-executable code is further configured to filter out skeletons near the active scan line, wherein the active scan line separates the active and deactivated regions.

[0020] In some embodiments, the computer-executable code is configured to track skeletons identified in an image via bipartite graph matching.

[0021] In some embodiments, the neural network is a convolutional neural network (CNN), and the animal is a pig.

[0022] In some embodiments, the imaging device is mounted on a rail above the building and running from one end of the building to the other.

[0023] In some embodiments, the neural network is trained on a remote computing device, which can move with the imaging device and is an embedded device. In other words, the neural network can be trained on the computing device of the pig counting system, or on some other computer, and then transmitted to the computing device of the pig counting system.

[0024] In some embodiments, the imaging device includes multiple cameras fixed to the roof of the house such that the field of view of the cameras covers the entire interior area of ​​the house.

[0025] In some embodiments, computing and imaging devices are mounted on the drone.

[0026] In some aspects, this disclosure relates to a method for counting animals in a dwelling. In some embodiments, the method includes:

[0027] The computing device receives multiple images captured by the imaging device from one end of the building toward the opposite end of the building;

[0028] The key points in each image are detected using a neural network by a computing device;

[0029] The skeleton of an animal is identified by a computing device by associating multiple key points in an image, where each skeleton corresponds to one animal;

[0030] The skeleton identified in the image is tracked by a computing device to obtain the skeleton's trajectory;

[0031] The computing device divides each image into active and deactivated regions;

[0032] The computing device assigns a first spatial value to the skeleton in the active region and a second spatial value to the skeleton in the deactivated region;

[0033] The computing device provides a trajectory count for each trajectory by summing the first spatial value and the second spatial value of a corresponding skeleton in a given trajectory; and

[0034] The counting of the animals is obtained by adding the track counts together using a computing device.

[0035] In some embodiments, each skeleton includes at least one of five types of keypoints, namely, head keypoints corresponding to the animal's head, neck keypoints corresponding to the animal's neck, back keypoints corresponding to the animal's back, waist keypoints corresponding to the animal's waist, and tail keypoints corresponding to the animal's tail. In some embodiments, each detected keypoint includes a classification and a vector, wherein the classification indicates which of the five types of keypoints the detected keypoint belongs to, and the vector indicates a direction toward at least one detected keypoint belonging to the same skeleton as the detected keypoint.

[0036] In some embodiments, the first space value is 0 and the second space value is 1.

[0037] In some embodiments, the size of the active region and the size of the deactivated region are substantially equal, and the active region is located in front of the deactivated region in the direction of movement of the imaging device.

[0038] In some embodiments, the method further includes filtering out skeletons near active scan lines by a computing device, wherein the active scan lines separate active and deactivated regions.

[0039] In some embodiments, the step of tracking the skeleton identified in the image is performed by bipartite graph matching.

[0040] In some embodiments, the neural network is a convolutional neural network (CNN), and the animal is a pig.

[0041] In some embodiments, the imaging device is mounted on a rail above the building, running from one end of the building to the other. The neural network is trained in a remote computing device, which is movable along with the imaging device and is an embedded device.

[0042] In some respects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The computer-executable code, when executed at a processor of a computing device, is configured to perform the methods described above.

[0043] These and other aspects of this disclosure will become clear from the following description of preferred embodiments in conjunction with the accompanying drawings and descriptions, although variations and modifications therein may affect the spirit and scope of the novel concept of this disclosure. Attached Figure Description

[0044] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts of the embodiments, in which:

[0045] Figure 1 A pig counting system according to certain embodiments of the present disclosure is schematically depicted;

[0046] Figure 2 A pig counting system installed in a pigsty according to certain embodiments of the present disclosure is schematically depicted;

[0047] Figure 3 A fully convolutional neural network according to certain embodiments of the present disclosure is schematically depicted;

[0048] Figure 4 The skeleton of a pig, derived from a key point heatmap and a vector map according to certain embodiments of the present disclosure, is illustrated schematically.

[0049] Figure 5 Skeleton tracking according to certain embodiments of this disclosure is illustrated schematically;

[0050] Figure 6A The field of view of a video image according to certain embodiments of the present disclosure is schematically shown;

[0051] Figure 6B Spatial codes for two pig skeletons in consecutive video images according to certain embodiments of the present disclosure are schematically depicted;

[0052] Figures 7A to 7G Spatial codes of tracked pigs according to certain embodiments of this disclosure are schematically depicted;

[0053] Figure 7H The summation of two consecutive time codes according to certain embodiments of the present disclosure is illustrated schematically;

[0054] Figure 8A and 8B A process for counting pigs according to certain embodiments of the present disclosure is illustrated schematically;

[0055] Figure 9 A method for counting pigs according to certain embodiments of the present disclosure is illustrated schematically;

[0056] Figure 10A and Figure 10C The diagram schematically illustrates a top-down method for detecting pigs according to certain embodiments of the present disclosure; and

[0057] Figure 10B and Figure 10D A bottom-up method for detecting pigs according to certain embodiments of the present disclosure is illustrated schematically. Detailed Implementation

[0058] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the accompanying drawings, the same numerals indicate the same parts throughout the set of drawings. Unless the context clearly requires otherwise, the terms “a,” “an,” and “the” as used herein and throughout the claims have the meaning of the plural. Furthermore, as used in the description and claims of this disclosure, unless the context clearly requires otherwise, “in” has the meaning of “in” and “on”. And, for the convenience of the reader, headings or subheadings may be used in the specification without affecting the scope of the present disclosure. In addition, some terms used in this specification are given more specific definitions below.

[0059] The terms used in this specification generally have their ordinary meaning in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe this disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of this disclosure. It will be understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms may be used for any one or more terms discussed herein, and have no particular significance in whether a term is elaborated or discussed herein. This disclosure provides synonyms for certain terms. The use of one or more synonyms does not preclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is merely illustrative and in no way limits the scope or meaning of this disclosure or any exemplary terms. Likewise, this disclosure is not limited to the various embodiments given in this specification.

[0060] Unless otherwise defined, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having the same meaning as they have in the context of the relevant technology and this disclosure, and should not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0061] As stated herein, “approximately,” “roughly,” “substantially,” or “approximately” should generally mean within 20%, preferably within 10%, and more preferably within 5% of a given value or range. The values ​​given herein are approximate, meaning that the terms “approximately,” “roughly,” “substantially,” or “approximately” can be inferred unless explicitly stated otherwise.

[0062] As stated in this article, "multiple" refers to two or more.

[0063] As stated in this article, the terms “including,” “contains,” “carries,” “has,” “includes,” “involves,” etc., should be understood as open-ended, meaning including but not limited to.

[0064] As described herein, at least one of the phrases A, B, and C should be interpreted as representing logic (A or B or C) using non-exclusive logical OR. It should be understood that one or more steps within the method may be performed in different orders (or simultaneously) without altering the principles of this disclosure. As described herein, the term "and / or" includes any and all combinations of one or more of the related listed items.

[0065] As described herein, the term "module" can refer to, for example, an application-specific integrated circuit (ASIC) in a system-on-a-chip; electronic circuitry; combinational logic circuitry; field-programmable gate array (FPGA); a (shared, dedicated, or group) processor that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above. The term "module" can also include (shared, dedicated, or group) memory storing code executed by the processor.

[0066] The term "code" as used herein can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared" as used above means that some or all of the code from multiple modules can be executed using a single (shared) processor. Furthermore, some or all of the code from multiple modules can be stored in a single (shared) memory. The term "group" as used above means that some or all of the code from a single module can be executed using a group of processors. Furthermore, a group of memory can be used to store some or all of the code from a single module.

[0067] As described herein, the term "interface" generally refers to a communication tool or device used at the point of interaction between components to perform data communication between components. Generally, an interface can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. For example, the components communicating with the interface can be multiple components of a computer system or peripheral devices.

[0068] This disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, shown in solid boxes, and virtual software components, shown in dashed boxes. Those skilled in the art will understand that, unless otherwise stated, these computer components may be implemented in the form of software, firmware, or hardware components, or combinations thereof, but are not limited to these forms.

[0069] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0070] The present disclosure is described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0071] In some aspects, this disclosure relates to automated pig counting and monitoring using computer vision (CV) technology, a promising approach to support intensive pig farming management while reducing costs. In some embodiments, this disclosure has a field of view covering a large area, can handle situations where pigs frequently enter and exit the camera's field of view, retains detailed individual information about the pigs, uses accurate location information of the pigs, correlates targets across time, is suitable for flexible, moving targets, and is applicable to indoor scenes with occlusion and overlap.

[0072] Figure 1 A pig counting system according to certain embodiments of the present disclosure is schematically depicted. For example... Figure 1As shown, system 100 includes a robotic device 110, a computing device 130 on which the robotic device 110 is mounted, an imaging device 170 attached to and communicating with the computing device 130, and an optional guide rail 190 for movement of the robotic device 110 thereal. In some embodiments, the computing device 130 may be a dedicated computer with limited processing, memory, and storage capacity, which reduces the cost of system 100 while still having sufficient power to perform the functions of system 100. In some embodiments, the computing device 130 may also be a server computer, cluster, cloud computer, or general-purpose computer located remotely from system 100 and communicating with the imaging device 170. In some embodiments, the computing device 130 is an embedded device. In some embodiments, communication is via a network, which may be a wired or wireless network and may take various forms, such as public and private networks; or via a non-transitory computer medium, including but not limited to flash drives, USB drives, hard disk drives, floppy disks, SD cards, optical drives, or any other portable data storage medium. In some embodiments, computing device 130 provides a user interface for configuring parameters of robotic device 110, computing device 130, and imaging device 170, configuring or managing communication between devices 110, 130, and 170, and / or presenting counting results to pig farm workers. In some embodiments, robotic device 110, computing device 130, and imaging device 170 are integrated together and move together along guide rail 190.

[0073] like Figure 1 As shown, computing device 130 may include, but is not limited to, processor 132, memory 134, and storage device 136. In some embodiments, computing device 130 may include other hardware and software components (not shown) to perform its respective tasks. Examples of such hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0074] Processor 132 may be a Central Processing Unit (CPU) for controlling the operation of computing device 130. Processor 132 may execute the operating system (OS) or other applications of computing device 130. In some embodiments, computing device 130 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. Memory 134 may be volatile memory, such as random-access memory (RAM), for storing data and information during the operation of computing device 130. In some embodiments, memory 134 may be an array of volatile memory. In some embodiments, computing device 130 may run on more than one memory 134. Storage device 136 is a non-volatile data storage medium for storing the OS (not shown) and other applications of computing device 130. Examples of storage device 136 may include non-volatile memory such as flash memory, memory cards, USB drives, hard disk drives, floppy disks, optical drives, solid-state drives (SSDs), or any other type of data storage device. In some embodiments, storage device 136 may be local storage, remote storage, or cloud storage. In some embodiments, computing device 130 may have multiple storage devices 136, which may be the same type of storage device or different types of storage devices, and applications of computing device 130 may be stored in one or more storage devices 136 of computing device 130. In some embodiments, computing device 130 is a cloud computer, and processor 132, memory 134, and storage devices 136 are shared resources provided on demand via the Internet. In some embodiments, computing device 130 is an embedded system designed for pig counting and may include a microprocessor with integrated memory and peripheral interfaces, or may include a general-purpose microprocessor.

[0075] like Figure 1As shown, storage device 136 includes a pig counting application 138. Pig counting application 138 provides pig counting services to pig farms and is capable of automatic pig counting based on video frames or video images. Pig counting application 138 includes an image capture module 140, a neural network 142, a keypoint association module 144, a tracking module 146, a spatial encoding module 148, a temporal filtering module 150, a user interface 152, and a robot control module 154. In some embodiments, pig counting application 138 may include other applications or modules necessary for its operation. It should be noted that each module is implemented by computer-executable code or instructions, which together constitute an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code, and computing device 130 may be implemented by a simple circuit board with other accessory components mounted on the board. In some embodiments, some or all modules of pig counting application 138 may be located at a remote computing device or distributed in the cloud.

[0076] The video capture module 140 is used to instruct the imaging device 170 to capture multiple sequences of images or video frames in the pig house of the pig farm when receiving an instruction or when executing the pig counting application 138, and to send the captured video frames to the neural network 142. Figure 2 A system 100 installed in a pigsty 200 is schematically shown. The pigsty 200 can be a barn or large enclosure for a herd of pigs. The herd can consist of piglets from birth to weaning, nursery pigs from weaning to approximately 6-8 weeks post-weaning, or growing-finishing pigs from nursery to full maturity, typically lasting 16-17 weeks. The pigsty 200 can have wooden, concrete, or stone floors, and the ceiling can allow some sunlight or be completely shaded. In some embodiments, the pigsty 200 of this disclosure has an enclosed space, preventing pigs from freely leaving the pigsty 200. Figure 2As shown, guide rail 190 can be mounted on the ceiling of pigpen 200 and has a first end 192 and a second end 194. Robot device 110 can move back and forth along guide rail 190. Imaging device 170 is attached below robot device 110. When robot device 110 is in position 196, imaging device 170 has a field of view 202 as shown by the dashed rectangle. Note that the length direction or "X" direction of field of view 202 is a portion of the length of pigpen 200, and the width direction or "Y" direction of field of view 202 is substantially the same as the width of pigpen 200. With this design, the field of view 202 of imaging device 170 will completely cover the width direction of pigpen. Active scan line 204 divides field of view 202 into two zones: a deactivated zone 206 on the rear side of the movement direction of robot device 110, and an activated zone 208 on the front side of the movement direction of robot device 110. In some embodiments, active scan line 204 is located substantially directly below imaging device 170. In some embodiments, the image captured by the imaging device 170 may be a red-green-blue (RGB) image or a black-and-white image.

[0077] The neural network 142 is used to perform neural network operations on the image to detect key points in the image when a captured image is received from the image capture module 140, and to send the detected key points to the key point association module 144. In some embodiments, the neural network 142 is a bottom-up detection convolutional neural network (CNN). Figure 3 A fully convolutional network 300 is schematically depicted for detecting visible keypoints in images belonging to each pig. Figure 3 As shown, CNN300 includes convolutional layers, convolutional blocks, max-pooling layers, and upsampling layers. Depthwise separable convolutions are used as basic building blocks to reduce computational cost. Position-preserving max-pooling is used to improve positional accuracy; it preserves the indices of the encoder's max-pooling layers and passes these indices to the corresponding upsampling layers in the decoder.

[0078] Training data is provided for training neural network 300. In some embodiments, the training data includes multiple images of pigs in one or more pigpens. Detection of rigid objects in images is generally reliable. However, pigs are often active in pigpens and are flexible, making it difficult to accurately detect and distinguish them. To address this issue, in some embodiments, this disclosure uses associated keypoints to represent pigs instead of defining bounding boxes and extracting features from them. In some embodiments, this disclosure uses five specific keypoints to identify each pig, corresponding to the head, neck, back, loin, and tail. In some embodiments, the head and tail are labeled as the same category for training neural network 300, and the neck and loin are labeled as the same category for training neural network 300. To further improve detection accuracy, some embodiments of this disclosure also include associations between keypoints. For example, for the five keypoints of a pig, the distance and relative angle between adjacent keypoints are also recorded. The distance between keypoints, especially between adjacent keypoints, can be measured using images. The direction between keypoints, especially between adjacent keypoints, can be represented by a vector from one keypoint to another. Using the labels of the location and orientation of five key points of a pig in an image, a neural network 142 can be effectively trained to infer the relationship between adjacent key points.

[0079] Then, a well-trained neural network 142 can be used to predict the key points of the pig in each captured image. Figure 4 The diagram illustrates the prediction of keypoints and the association of keypoints to obtain the skeleton. For example... Figure 4As shown, when the captured frame 402 is used as input to the neural network 142, the neural network 142 generates a keypoint heatmap 404 and a vector map 406 based on the captured frame 402. As shown in 404, each pig is marked with five keypoints. By defining different parts of the pig using different keypoints, this disclosure can be used to easily distinguish one pig from other pigs. For example, if two head keypoints and two neck keypoints are close to each other, the two head keypoints are unlikely to belong to the same pig, and the two neck keypoints are also unlikely to belong to the same pig. Furthermore, head-neck pairs can be further determined based on the distance and vector direction of the keypoints, and optionally based on back keypoints, waist keypoints, and tail keypoints in neighboring regions. Specifically, the vector map 406 can be used to determine the relationships between adjacent keypoints. In some embodiments, the heatmap may include six channels: five channels for each keypoint type and one channel for the background. In some embodiments, the head and tail are defined as the same type, and the neck and waist are defined as the same type, thus simplifying the five types to three: head / tail keypoints, neck / waist keypoints, and back keypoints. Therefore, the heatmap presents four channels: three for each keypoint type and one for the background. The heatmap prediction is then represented as a pixel-wise multi-classification problem, and each pixel is learned by the neural network 142 to belong to either one of the keypoint types or the background.

[0080] The keypoint association module 144, after receiving the heatmap and vector map of keypoints from the neural network 142, associates the keypoints of each pig in the image to form a skeleton, and sends the skeleton to the tracking module 146. For example... Figure 4 As shown, the skeletons 408 of pigs 1, 2, 3, and 4 are derived from the keypoint heatmap 404 and the vector map 406. In some embodiments, keypoints belonging to the same pig are associated when using the distance and vector direction between keypoints. Therefore, each pig is represented by five associated keypoints. In some embodiments, part of the pig in the image may be occluded, and there may be fewer than five associated keypoints for a pig.

[0081] In some embodiments, due to the unknowable instances of predicted keypoints on the heatmap, a unique instance ID can be assigned to each detected keypoint so that the present disclosure can "connect" "points" belonging to the same individual instance. To this end, the present disclosure adds a separate two-channel output of an offset field to the neural network, which indicates the displacement from a given keypoint to its parent node in the skeleton. During the testing phase, an iterative greedy algorithm is used to associate the predicted keypoints. Alternatively, the present disclosure searches for the best candidate parent node for all predicted keypoints and removes the remaining keypoints from their candidate sublists until no better hypothesis is found. The best candidate parent node is defined as the keypoint in the correct class that best matches the predicted offset vector. The Euclidean distance between the predicted offset and the actual offset is used to measure the degree of matching. In some embodiments, the mid-point or back point of the pig is defined as the root keypoint, and the back keypoint is the parent node of the corresponding neck keypoint and waist keypoint. Subsequently, the neck keypoint is the parent node of the head keypoint, and the waist keypoint is the parent node of the tail keypoint. Five key features of a pig, along with the father / son relationship, define the pig's skeleton.

[0082] The tracking module 146 is used to track the skeleton of a pig in the received sequence of images and send the tracked skeleton to the spatial encoding module 148. In some embodiments, an efficient online tracking method is used to associate pig keypoints in time. This method takes a combination of pig keypoints (skeletons) for a single frame as input and then assigns a unique identifier (ID) number to each pig in each frame. In some embodiments, the problem is formulated as an energy maximization problem based on bipartite graph matching. The estimated pig candidate C_t at frame t is then associated with the previous pig candidate C_(t-1) at frame t-1 by bipartite graph matching. In some embodiments, this bipartite graph matching problem is solved using the Hungarian algorithm. Figure 5 The process of skeleton tracking is illustrated schematically. For example... Figure 5 As shown, the skeletons in each frame t, t-1, t-2, ... are derived from the corresponding vector graph 502 and heatmap 504 via association 506. Each image includes detected keypoints, and keypoints belonging to the same group are assigned specific skeleton IDs. The skeletons of the same pig in multiple images 508 are then tracked to obtain the trajectory of each tracked pig.

[0083] The spatial coding module 148 is used to assign a spatial code to each pig in each image after receiving the tracked image, and send the spatial code to the temporal filtering module 150. Figure 6A The diagram schematically depicts the field of view of an image. Figure 6B The spatial codes of two pigs in a series of images are schematically depicted. For example... Figure 6A As shown, for each image, the spatial coding module 148 uses an active scan line 602 to divide the image into an active region 604 and a deactivated region 606. In some embodiments, the active scan line 602 is substantially perpendicular to the direction of movement of the robotic device 110. The active scan line 602 is located approximately at the center of the field of view and perpendicular to the direction of movement, such that the active region 604 and the deactivated region 606 are the same or similar in size. The active scan line 602 is stationary in a single image but moves with the robotic device 110 to scan the entire pigpen. In other words, the active scan line 602 moves between different images. When the skeleton of a pig is located in the active region 604 of an image, the skeleton is assigned a corresponding code value, such as code value 0. When the skeleton of a pig is located in the deactivated region 606 of an image, the skeleton is assigned a corresponding code value, such as code value 1. Deactivation area 606 indicates that all candidate pigs in deactivation area 606 have been counted using pig counting application 138, and candidate pigs in activation area 604 will be counted when active scan line 602 scans through them. Figure 6B As shown, the time-lapse images display two tracked pigs. The pig on the left is in the deactivated region in the time-lapse image, while the pig on the right is in the deactivated region in the top image and in the activated region in the bottom three images. Because the robotic device 110 moves during image capture, the deactivated and activated regions are dynamic in the images.

[0084] The temporal filtering module 150 is used to filter the temporal skeleton to count pigs upon receiving the spatial code of a pig skeleton in an image, and then send the count to the user interface 152. In some embodiments, upon receiving the spatial code of a pig in an image, the temporal filtering module 150 is used to combine the spatial code values ​​of the same pig in the sequence of images to generate a spatial code list. A pig or a trajectory of that pig has a spatial code list, and each spatial code corresponds to the state of the pig at a given time point. Figure 7A The code values ​​of pigs at time points t-6, t-5, t-4, t-3, t-2, t-1, and t are schematically shown. The code values ​​for pigs located in the activation region at time points t-6, t-5, t-4, and t-3 are 0, and the code values ​​for pigs located in the deactivation region at time points t-2, t-1, and t are 1. The first-order difference between each of the following pairs of pigs is 0: between t-6 and t-5, between t-5 and t-4, between t-4 and t-3, between t-2 and t-1, and between t-1 and t. The first-order difference between t-3 and t-2 is 1. Therefore, the generated list of time code values ​​for pigs from t-6 to t is [0,0,0,0,1,1,1], with first-order differences of [0,0,0,1,0,0]. The time filtering module 150 defines the final count for this trajectory as the sum of the first-order differences of the time codes. Therefore, the pig's final count is 1, indicating that the pig was scanned once when it moved from the active area to the deactivated area. Therefore, due to... Figure 7AThe code value for the pig shown should be incremented by 1 to represent the total number of pigs. Similarly, Figure 7B The image shows the trajectory of a pig with the code [1,0,0,0,1,1,0,0], and the sum of the first-order differences indicates a count of -1. This means that the previously counted pig has moved from the scanned area (deactivated area) to the area to be scanned (activated area). Therefore, the total count should be decremented by 1. This design allows the algorithm to avoid false counts caused by pigs entering or leaving the camera's field of view and moving from one area to another. Figure 7C -G shows examples when the pig track counts are -1, 0, 1, 0, and 0. Figure 7F This indicates the pig's trajectory that never crossed the active scan line 702. Figure 7D and Figure 7G This indicates that the trajectory begins and ends in the same active or deactivated region. Figure 7H The sum of two consecutive timecodes is schematically shown, where the first-order difference of the consecutive timecodes results in a count. These examples demonstrate that certain methods of this disclosure are not affected by tracking failures occurring only in a single region (e.g., a trajectory being broken down into multiple trajectories due to occlusion). In some embodiments, a low-pass filter with a window size of 5 is applied before calculating the first-order difference. This low-pass filtering step is designed to avoid trajectory jitter near the active scan line. The final count result for the entire sequence of images or videos will also be added to the number of candidate objects detected in the deactivated region of the starting image and the number of candidate objects detected in the active region of the ending image. In some embodiments, when the imaging device 170 captures an image at the end of the scanning process, the pigs in the active region are counted directly based on the number of skeletons in the active region, rather than using the timecode for counting. In other words, the total number of pigs in the video is the same as the first-order difference of the timecode and the number of skeletons in the active region of the last frame. In an alternative embodiment, when the imaging device 170 captures an image at the end of the scanning process, the final image of the video can be entirely defined as a deactivated region so that all pigs in the image can be counted. After the robotic device 110 moves from one end of the pigsty 200 to the other end, the entire pigsty area is scanned, and the total number corresponds to the total number of pigs in the pigsty 200.

[0085] User interface 152 is used to send or present the counts to a user upon receiving a count of pigs in a time-lapse image. In some embodiments, user interface 152 may include a screen displaying the counts. In some embodiments, user interface 152 may send the counts to a user via a wireless or wired network, or to a cloud server accessible to the user. In some embodiments, user interface 152 may also provide a user interface for defining image capture parameters of imaging device 170, operating parameters of robotic device 110, and processing parameters of pig counting application 138.

[0086] The robot control module 154 is used to control the movement of the robot device 110. In some embodiments, the movement of the robot device 110 and the image capture of the imaging device 170 are coordinated by the pig counting application 138, and the movement of the robot device 110 from one end of the guide rail 190 to the other is accompanied by the image capture of the imaging device 170.

[0087] Imaging device 170 is used to capture video images of pigsty 200 and send the captured video images to pig counting application 138. In some embodiments, imaging device 170 may be an RGB camera or a grayscale camera. In some embodiments, imaging device 170 may also be an RGBD camera with depth detection capabilities.

[0088] Guide rail 190 is mounted on the ceiling or frame of the top space of pigsty 200. Robotic device 110 may be mounted on guide rail 190 and guided to move along guide rail 190. In some embodiments, robotic device 110 includes a motor to move itself along guide rail 190. In other embodiments, a motor may also be mounted on guide rail 190, and robotic device 110 may move passively along guide rail 190.

[0089] In some embodiments, system 100 may also include other components as needed. For example, when training neural network 142 in the field, storage device 136 may also include a database for training neural network 142; when local display is required, system 100 may also include a display device connected to user interface 152 for displaying pig counting results. In some embodiments, not... Figure 1 All components shown are necessary for pig counting. For example, pig counting application 138 may not include user interface 152, and the counting results may be displayed directly on the display after the time filtering module 150 counts the pigs, or the counting results may be transmitted to a remote device so that the user can access the counting results remotely.

[0090] Figure 8A and 8B A process for counting pigs according to certain embodiments of this disclosure is schematically depicted. Figure 8A and 8BAs shown, the inspection robot can move on a guide rail above the pigpen. The robot's camera captures video, with video frames t, t-1, t-2, ..., used as input to a neural network. The neural network generates a vector field and a heatmap for each input frame and correlates the vector field and heatmap for each frame to obtain the skeleton of each pig in that frame. The skeletons of pigs are tracked in consecutive frames, with each skeleton corresponding to a tracking trajectory. For each trajectory, the skeleton in each frame is encoded based on whether the skeleton is located in an active or inactive region. Spatial encoding is recorded and filtered, and a count of 0 or 1 is derived for each trajectory. The sum of the counts is the total number of pigs in the pen.

[0091] Figure 9 A method for counting pigs according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, the method is performed by... Figure 1 The robot device 110 shown is implemented. In some embodiments, Figure 9 The method 900 shown corresponds to the function of pig counting application 138. It should be specifically noted that, unless otherwise stated in this disclosure, the steps of the method may be in a different order and are not limited to this. Figure 9 The order shown.

[0092] like Figure 9 As shown, at step 902, when the pig counting application 138 is activated, the robot device 110 moves along the guide rail 190 from one end to the other, the video capture module 140 instructs the imaging device 170 to capture an image of the pig house 200, and sends the captured image to the neural network 142.

[0093] At step 904, upon receiving a captured image, neural network 142 processes each image to detect keypoints and sends the detected keypoints to keypoint association module 144. In some embodiments, for each detected keypoint in each frame, the detected features of the keypoint include not only the position of each keypoint but also the vector direction of the keypoint.

[0094] At step 906, upon receiving detected keypoints, the keypoint association module 144 combines the relevant keypoints in each image into a skeleton and sends the skeleton to the tracking module 146. Each skeleton corresponds to a pig in the frame. In some embodiments, the skeleton is determined based on the distance and vector direction between keypoints. In some embodiments, each skeleton may include five keypoints corresponding to the pig's head, neck, back, waist, and tail.

[0095] At step 908, when a skeleton in a frame is received, the tracking module 146 tracks the skeleton in the image to form a skeleton trajectory and sends the trajectory to the spatial encoding module 148. Specifically, the tracking module 146 assigns a tracking ID or skeleton ID to each skeleton in the frame and tracks the movement of the skeleton across multiple sequence frames. In some embodiments, skeletons in a sequence of images are determined to have the same tracking ID or skeleton ID based on the skeleton's position, the distribution of keypoints in the skeleton, and the vector orientation of the keypoints in the skeleton.

[0096] At step 910, after receiving the trajectory of the skeleton in the sequence frame, the spatial encoding module 148 provides a spatial code for each skeleton in each frame and sends the spatial code to the temporal filtering module 150. Specifically, for each frame, the spatial encoding module 148 divides the field of view into an active region and a deactivated region. The frame covers the entire width of the pigpen. The active region is located in front of the robot's direction of movement; pigs in the active region are counted and assigned a spatial code value of 0. The deactivated region is located behind the active region; pigs in the deactivated region have been counted and assigned a spatial code value of 1.

[0097] At step 912, upon receiving the spatial code of the skeleton in the frame, the time filtering module 150 processes the spatial value to obtain the pig count in the pigpen and sends the count to the user interface 152. Each skeleton's trajectory corresponds to a list of spatial codes, which in turn corresponds to one or more sequence frames containing that skeleton. The count for each trajectory is the sum of the first-order differences of the spatial codes, and the sum of the counts for all trajectories is the final pig count. In some embodiments, pigs may be lost from some images and may have broken trajectories. Broken trajectories may include multiple trajectories determined by the pig counting application 138. In other words, the same pig may correspond to multiple trajectories, and the pig counting application 138 of this disclosure cannot distinguish that multiple trajectories belong to the same pig. Through the above spatial code specification and counting calculation, although the application can identify multiple trajectories as corresponding to different pigs, the sum of the counts of multiple trajectories will still match the pig count, either being 0 or 1.

[0098] At step 914, when the final count of pigs in pigsty 200 is received, user interface 152 displays the count for user viewing or transmits the count to another device accessible to the user.

[0099] In some embodiments, the above method can count not only pigs in a pigsty, but also other animals in an enclosed space.

[0100] In another aspect, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The code, when executed at a processor of a computing device, is used to perform the methods described above. In some embodiments, the non-transitory computer-readable medium may include, but is not limited to, any physical or virtual storage medium. In some embodiments, the non-transitory computer-readable medium may be implemented as follows: Figure 1 The storage device 136 of the computing device 130 shown.

[0101] In some aspects, this disclosure can also be implemented by using a top-down approach instead of a bottom-up approach for pig detection. In embodiments, this disclosure uses detectors such as Single Shot Detector (SSD), YOLO (You Only Look Once), or Faster-RCNN based convolutional neural networks to detect bounding boxes. However, these methods are less accurate or efficient than the embodiments described above. These methods first propose using bounding boxes to detect candidate locations, and then classify each box as either a real or non-real target. In some embodiments, Non-Maximum Suppression (NMS) is used as a post-processing method to significantly reduce false positives by removing bounding boxes that have a high overlap ratio (cross-union ratio). Nevertheless, in this application, using bounding boxes to locate pigs is suboptimal. Deformable and elongated elliptical pigs are particularly challenging for bounding box-based methods in crowded scenes. Figure 10A and Figure 10C As shown, the bounding boxes around two adjacent pigs have a very high overlap rate, and this ambiguity tends to confuse neural network training. Furthermore, for inference, the NMS post-processing step forces the detector to select only one bounding box in these highly overlapping cases, leading to false negatives. Compared to bounding boxes, such as Figure 10B and Figure 10D The pig skeleton shown, defined by key points, is better suited for distinguishing pigs within a herd.

[0102] In some embodiments, the system may not require the installation of the guide rail 190; the robotic device 110 and the imaging device 170 can be mounted on a drone. The drone can fly from one end of the pigsty to the other, allowing the imaging device 170 to capture video images of the pigsty, and the robotic device 110 to process the captured video images to obtain a count of the pigs in the pigsty.

[0103] In some embodiments, instead of moving the imaging device 170 from one end of the pigsty to the other, the system can install multiple imaging devices in different areas so that the images captured by the multiple imaging devices cover the entire pigsty. However, due to the difficulty and computational resources required to integrate the captured images, using multiple fixed imaging devices is less efficient than using the movable imaging device 170.

[0104] Certain embodiments of this disclosure offer particular advantages. First, pig counting is performed automatically based on captured video images, saving the cost of pig counting using other methods. Second, detecting pigs in captured video frames does not require bounding box detection, avoiding situations where a single bounding box covers more than one pig due to overcrowding in the pigpen. In contrast, this disclosure provides a bottom-up solution where keypoints are directly detected, and the pig's skeleton is derived from the combined keypoints. Third, the combined keypoints take into account the pig's flexibility and rapid movement within the pigpen. Specifically, this disclosure considers the distance and vector direction between keypoints to combine keypoints belonging to pigs, making pig identification more accurate. Fourth, each video frame is divided into active and deactivated regions, and each pig skeleton position in the pig's trajectory is assigned a spatial code value based on the region to which the skeleton belongs. The spatial code values ​​for each trajectory are summed using first-order differences to obtain an accurate count. Therefore, a pig moving from one region to another or leaving a region does not affect accurate counting, and a rapidly moving pig moving from one region to another will not be counted multiple times. Furthermore, even if a pig is missing in several intermediate frames, the trajectory of that pig is treated as multiple separate trajectories belonging to different pigs, and the above count can still capture this situation and provide an accurate count of 0 or 1 for multiple trajectories.

[0105] The foregoing description of exemplary embodiments of this disclosure is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Many modifications and variations are possible in accordance with the foregoing teachings.

[0106] The embodiments were chosen and described to explain the principles of this disclosure and its practical application, thereby enabling others skilled in the art to utilize this disclosure and various embodiments, as well as various modifications suitable for the particular intended use. Alternative embodiments will become apparent to those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

[0107] References:

[0108] [1]TIAN MENGXIAO,HAO GUO,HONG CHEN,et al.,Automated pig countingusing deep learning,Computers and Electronics in Agriculture,2019,163,0168-1699.

[0109] [2]XU LIU,STEVEN W.CHEN,SHREYAS ADITYA,et al,Robust fruit counting:combining deep learning,tracking,and structure from motion,2018IEEE / RSJInternational Conference on Intelligent Robots and Systems(IROS),2018,arXiv:1804.00307.

[0110] [3]XU LIU,STEVEN W CHEN,CHENHAO LIU,et al.,Monocular camera basedfruit counting and mapping with semantic data association,IEEE Robotics andAutomation Letters,2019,4(3):2296–2303.

[0111] [4]JARROD C HODGSON,SHANE M BAYLIS,ROWAN MOTT,et al.,Precisionwildlife monitoring using unmanned aerial vehicles,Scientific reports,2016,6:22574.

[0112] [5]RIVAS A,CHAMOSO P,GONZALEZ-BRIONES A,CORCHADO JM,Detection ofcattle using drones and convolutional neural networks,Sensors,2018,18(7):2048.

[0113] [6]ANDREW G HOWARD,MENGLONG ZHU,BO CHEN,et al.,Mobilenets:efficientconvolutional neural networks for mobile vision applications,2017,arXiv:1704.04861.

[0114] [7]VIJAY BADRINARAYANAN,ALEX KENDALL,ROBERTO CIPOLLA,SegNet:a deepconvolutional encoder-decoder architecture for image segmentation,IEEEtransactions on pattern analysis and machine intelligence,2017,39(12):2481-95.

[0115] [8]WEI LIU,DRAGOMIR ANGUELOVE,DUMITRU ERHAN,et al,SSD:single shotmultibox detector,European conference on computer vision,2016,21-37,Springer,Cham,arXiv:1512.02325.

[0116] [9]JOSEPH REDMON,SANTOSH DIVVALA,ROSS GIRSHICK,ALI FARHADI,You onlylook once:unified,real-time object detection,Proceedings of the IEEEconference on computer vision and pattern recognition,2016,779-788.

[0117]

[10] ROSS GIRSHICK,Fast R-CNN,Proceedings of the IEEE internationalconference on computer vision,2015,1440-1448.

Claims

1. A system for counting animals in a dwelling, the system comprising a computing device and an imaging device, the computing device including a processor and a storage device storing computer-executable code, wherein, The computer-executable code, when executed at the processor, is configured to: Receive multiple images captured by the imaging device from one end of the building to the opposite end of the building; Use a neural network to detect key points in each image; The skeleton of an animal is identified by associating multiple key points in the image, where each skeleton corresponds to one animal; Track the skeleton identified in the image to obtain the trajectory of the skeleton; Each image is divided into an active region and a deactivated region, the size of the active region and the size of the deactivated region are substantially equal, and the active region is located in front of the deactivated region in the direction of movement of the imaging device; When the skeleton is located in the active region, a first spatial value is assigned to the skeleton in the active region, and when the skeleton is located in the deactivated region, a second spatial value is assigned to the skeleton in the deactivated region. A trajectory count is provided for each trajectory by summing the first spatial value and the second spatial value of a corresponding skeleton in a given trajectory; and The count of the animal is obtained by adding the track counts of the tracks together.

2. The system according to claim 1, wherein, Each skeleton includes at least one of five types of keypoints, namely, head keypoints corresponding to the animal's head, neck keypoints corresponding to the animal's neck, back keypoints corresponding to the animal's back, waist keypoints corresponding to the animal's waist, and tail keypoints corresponding to the animal's tail. as well as Each detected keypoint includes a classification and a vector, wherein the classification indicates which of the five keypoint types the detected keypoint belongs to, and the vector indicates a direction toward at least one detected keypoint belonging to the same skeleton as the detected keypoint.

3. The system according to claim 1, wherein, The first space value is 0, and the second space value is 1.

4. The system according to claim 1, wherein, The computer-executable code is also configured to filter out skeletons near the active scan line, which separates the activation region from the deactivation region.

5. The system according to claim 1, wherein, The computer-executable code is configured to track skeletons identified in the image through bipartite graph matching.

6. The system according to claim 1, wherein, The neural network is a convolutional neural network (CNN), and the animal is a pig.

7. The system according to claim 1, wherein, The imaging device is mounted on a rail above the building, running from one end of the building to the other end.

8. The system according to claim 7, wherein, The neural network is trained in a remote computing device that can move with the imaging device and is an embedded device.

9. The system according to claim 1, wherein, The computing device and the imaging device are mounted on the drone.

10. A method for counting animals in a dwelling, comprising: The computing device receives multiple images captured by the imaging device from one end of the building to the opposite end of the building; The computing device uses a neural network to detect key points in each image; The computing device identifies the skeleton of the animal by associating multiple key points in the image, wherein each skeleton corresponds to one animal; The computing device tracks the skeleton identified in the image to obtain the trajectory of the skeleton; The computing device divides each image into an active region and a deactivated region, the size of the active region and the size of the deactivated region being substantially equal, and the active region being located in front of the deactivated region in the direction of movement of the imaging device; When the skeleton is located in the active region, the computing device assigns a first spatial value to the skeleton in the active region, and when the skeleton is located in the deactivated region, it assigns a second spatial value to the skeleton in the deactivated region. The computing device provides a trajectory count for each trajectory by summing the first spatial value and the second spatial value of a corresponding skeleton in a given trajectory; and The counting of the animal is obtained by adding the trajectory counts of the trajectory together by the computing device.

11. The method according to claim 10, wherein, Each skeleton includes at least one of five types of keypoints, namely, head keypoints corresponding to the animal's head, neck keypoints corresponding to the animal's neck, back keypoints corresponding to the animal's back, waist keypoints corresponding to the animal's waist, and tail keypoints corresponding to the animal's tail. as well as Each detected keypoint includes a classification and a vector, wherein the classification indicates which of the five keypoint types the detected keypoint belongs to, and the vector indicates a direction toward at least one detected keypoint belonging to the same skeleton as the detected keypoint.

12. The method according to claim 10, wherein, The first space value is 0, and the second space value is 1.

13. The method of claim 10, further comprising: The computing device filters out skeletons near the active scan line, wherein the active scan line separates the active region from the deactivated region.

14. The method of claim 10, wherein, The step of tracking the skeleton identified in the image is performed by bipartite graph matching.

15. The method according to claim 10, wherein, The neural network is a convolutional neural network (CNN), and the animal is a pig.

16. The method of claim 10, wherein, The imaging device is mounted on a rail above the building and running from one end of the building to the other end of the building. The neural network is trained in a remote computing device that can move with the imaging device and is an embedded device.

17. A non-transitory computer-readable medium storing computer-executable code, wherein, The computer-executable code, when executed at the processor of the computing device, is configured to: Receive multiple images captured by an imaging device from one end of the building to the opposite end of the building; Use a neural network to detect key points in each image; The skeleton of an animal is identified by associating multiple key points in the image, where each skeleton corresponds to one animal; Track the skeleton identified in the image to obtain the trajectory of the skeleton; Each image is divided into an active region and a deactivated region, the size of the active region and the size of the deactivated region are substantially equal, and the active region is located in front of the deactivated region in the direction of movement of the imaging device; When the skeleton is located in the active region, a first spatial value is assigned to the skeleton in the active region, and when the skeleton is located in the deactivated region, a second spatial value is assigned to the skeleton in the deactivated region. A trajectory count is provided for each trajectory by summing the first spatial value and the second spatial value of a corresponding skeleton in a given trajectory; and The count of the animal is obtained by adding the track counts of the tracks together.

18. The non-transitory computer-readable medium according to claim 17, wherein, Each skeleton includes at least one of five types of keypoints, namely, head keypoints corresponding to the animal's head, neck keypoints corresponding to the animal's neck, back keypoints corresponding to the animal's back, waist keypoints corresponding to the animal's waist, and tail keypoints corresponding to the animal's tail. Each detected keypoint includes a classification and a vector, wherein the classification indicates which of the five keypoint types the detected keypoint belongs to, and the vector indicates a direction toward at least one detected keypoint belonging to the same skeleton as the detected keypoint. Wherein, the first space value is 0, and the second space value is 1.

Images