Systems and methods for AI-based feeding in land-based fish farms

Abstract

Embodiments are described relating to an AI-based system and method for feed monitoring in land-based fish farms, where the system may include a feed camera attached to the discharge pipe for capturing a video feed having images of objects crossing the field of view of the feed camera at the time of acquisition of each image, and a dedicated computer executing a pellet tracking algorithm that employs a region of interest (ROI) proposal module, an ROI classification module, an ROI tracking module, and a trajectory classification module to count at least uneaten feed pellets. In at least one embodiment, an object enhancer may be attached to the discharge pipe opposite and within the field of view of the feed camera to provide a background so that objects in the acquired images have a definable surrounding.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 157,759, filed March 7, 2021, the entire contents of which are incorporated herein by reference in their entirety.

[0002] Generally, various embodiments are described herein relating to systems and methods for automated feeding in fish farms using artificial intelligence (AI). [Background technology]

[0003] The following paragraphs are provided as background to the present disclosure without any admission that what is described therein is prior art or part of the knowledge of the person skilled in the art.

[0004] Land-based aquaculture is a new approach to producing fish. Rather than utilizing the fish's natural habitat in the ocean and raising them in underwater cages, land-based aquaculture brings the fish onto land by replicating the fish's natural environment in tanks. The system constantly circulates and filters the water, attempting to obtain an optimal balance of environmental parameters to ensure the health and growth of the fish. This closed system approach has many benefits including the elimination of problems with pollution, environmental optimization, parasites, predators, and escapes. It also allows for farms to be located closer to consumer markets, reducing emissions by shortening transportation distances.

[0005] Despite the advanced knowledge required to create these land-based systems, feeding practices have remained relatively unevolved. While each farm approaches feeding differently, land-based farm operators do their best to understand the levels of waste feed. Many use qualitative monitoring of waste as a baseline (highly inaccurate), while others physically count waste feed pellets caught in filters (inaccurate, discrete, and / or time-consuming). After measuring the population's appetite through these methods, operators adjust their mechanical feeding systems based on the findings of waste feed. Due to the inefficiencies of the methods mentioned above, if waste is misjudged, farm operators have two distinct problems: (1) overfeeding fish, resulting in large amounts of fish feed being wasted, increasing costs and negative environmental impacts, or (2) underfeeding fish populations, resulting in stunted growth and loss of profitability. Finally, the constant movement of feed waste fluctuates environmental and water quality parameters, stressing the fish and filtration systems.

[0006] There are numerous challenges in developing automated feeding technology. The most notable is that accuracy is crucial. Small margins of error can result in instances of mass waste or starvation. Several solutions have been proposed in the subsea sector, all of which ignore the reality that using measures of pellet strength or precise measures of inconsistent cross-sections in space and time instances both provide only rough estimates of waste. Also, any method attempted to increase accuracy is by increasing the computational resources utilized, inflating costs beyond what is industrially feasible.

[0007] What is needed is a system and method for automated feeding in fish farms that addresses the problems and / or shortcomings discussed above. Summary of the Invention

[0008] Various embodiments of systems and methods for AI-based feeding in fish farms, and computer products for use therewith, are provided in accordance with the teachings herein.

[0009] According to one aspect of the present invention, a system for feed monitoring in a land-based fish farm having an aquarium containing fish is disclosed, the system comprising: a feed camera fixed to an outlet pipe coupled to the aquarium, the feed camera configured to capture a video feed comprising images of an interior of the outlet pipe including uneaten feed pellets and non-pellet objects flowing through the outlet pipe to an exit port and including objects comprising pellets and non-pellet objects crossing a field of view of the feed camera upon capture of each image; an object enhancer mounted on the outlet pipe opposite and within the field of view of the feed camera to provide a background to the feed camera, such that an object passing between the feed camera and the object enhancer and captured in at least one of the captured images of the video feed has a definable perimeter within the at least one captured image; and at least one processor, which when executed causes the at least one processor to monitor the feed at a given time interval. The method comprises: obtaining a number of images from the feed; pre-processing the images to obtain candidate regions of interest (ROIs), each corresponding to a unique object in the image, where the object is likely to be a feed pellet; classifying the candidate ROIs to obtain a pelletness score representing the probability that the object corresponding to the candidate ROI belongs to a feed pellet class; deciding whether to retain the candidate ROIs based on the likelihood that the retained candidate ROIs are likely to belong to the feed pellet class; assigning each of the retained candidate ROIs to a trajectory; classifying the trajectories into pellet trajectories or non-pellet trajectories; and generating in real time a pellet count representing a predetermined percentage of leftover feed pellets not consumed by the fish in the aquarium based on the pellet trajectories.

[0010] In at least one embodiment, the at least one processor is further configured to transmit the pellet count to the fish farm, wherein the pellet count is used by the fish feeder to generate a new amount of fish food to supply to the aquarium based at least in part on the pellet count.

[0011] In at least one embodiment, the at least one processor is further configured to generate a control signal based on the pellet count and send the control signal to a fish feeder in the fish farm to generate a new amount of fish feed to be fed to the aquarium.

[0012] In at least one embodiment, the at least one processor is further configured to determine a slope of the uneaten feed pellets over the time interval and generate an alert if the slope is greater than a slope threshold.

[0013] In at least one embodiment, the object enhancer is a high density polyethylene (HDPE) sheet that provides a contrasting background for objects passing between the feed camera and the object enhancer.

[0014] In at least one embodiment, at least one processor is configured to obtain candidate ROIs in a given image frame by performing background subtraction on the given image frame to identify any objects that are likely to change position in subsequent image frames, performing contour detection on the identified objects to define boundaries that are potential ROIs for each identified object, and filtering the potential ROIs to obtain candidate ROIs.

[0015] In at least one embodiment, at least one processor is configured to classify the candidate ROI by generating, for each candidate ROI, samples where a given sample includes image data of the corresponding candidate ROI, pre-processing the samples, and applying an ROI classifier to each pre-processed sample to generate a probability distribution having probabilities that the sample belongs to different classes.

[0016] In at least one embodiment, the classes include a pellet class, a non-pellet class, and an ambiguous class.

[0017] In at least one embodiment, the pelletness score represents the probability that a sample belongs to the pellet class.

[0018] In at least one embodiment, the at least one processor is configured to pre-process the samples by resizing the samples to have the same matrix size and performing mean-center normalization.

[0019] In at least one embodiment, at least one processor is configured to assign each of the candidate ROIs in one image frame to one of the trajectories by: (a) assigning the candidate ROI to another new trajectory if there is no active trajectory; (b) assigning the candidate ROI to the nearest active trajectory if the distance between the position of the candidate ROI and the position of the last ROI in the nearest active trajectory is less than a distance threshold; or (c) assigning the candidate ROI to a new active trajectory if there is an active trajectory if the distance between the position of the candidate ROI and the position of the last ROI in the nearest active trajectory is greater than a distance threshold.

[0020] In at least one embodiment, at least one processor is configured to classify the trajectories as pellet trajectories or non-pellet trajectories by identifying terminated trajectories including trajectories that have not been updated for one of the images in a given period of time, determining at least one trajectory measurement for each of the terminated trajectories, and determining which of the terminated trajectories belong to a pellet trajectory class based on the at least one trajectory measurement.

[0021] In at least one embodiment, the at least one processor is configured to determine which terminated trajectories belong to a pellet trajectory class by comparing the at least one trajectory measurement to a threshold or by providing the at least one trajectory measurement to a trajectory classifier.

[0022] In at least one embodiment, the at least one trajectory measurement includes at least one measure of the length of the completed trajectory, the distribution of pelletness scores of the ROI that comprises the completed trajectory, and / or the statistical distribution of one or more other filter values.

[0023] In at least one embodiment, the feed camera is coupled to an underside of the object enhancer and is spaced from the object enhancer such that a captured image shows at least a portion of the object enhancer.

[0024] In at least one embodiment, the system further comprises a bracket attached to the outlet pipe, and the object enhancer is mounted to the bracket in two or more positions to limit rotatability of the object enhancer and the feed camera during use.

[0025] In at least one embodiment, the object enhancer is slidably mounted adjacent the outlet port of the outlet conduit such that the object enhancer is movable away from the outlet port when an increased amount of water flow exits the outlet conduit.

[0026] In at least one embodiment, the feed camera is fixed to a bracket adjacent the outlet pipe and the object enhancer is slidably mounted on the post above the bracket.

[0027] In at least one embodiment, the feed camera is fixed to the outlet pipe and the object enhancer is slidably mounted adjacent to the outlet pipe so that it can move up and down as different amounts of water flow out of the outlet pipe.

[0028] In at least one embodiment, the object enhancer is slidably mounted adjacent to the discharge pipe and the feed camera is coupled to the object enhancer a fixed distance away from the object enhancer such that the object enhancer can move up and down as different amounts of water flow out of the discharge pipe and the feed camera is maintained at a fixed distance from the object enhancer.

[0029] In another broad aspect of the teachings herein, there is provided a computer-implemented method for feed monitoring in an aquarium of a land-based fish farm, the method comprising: reading image frames and associated metadata from a video feed acquired from a feed camera at a given time interval, the image frames acquired by the feed camera being images of leftover feed pellets and non-pellet objects being discharged from the aquarium; pre-processing the image frames to obtain candidate regions of interest (ROIs), each corresponding to a unique object in the image, where the object may be a feed pellet; classifying the candidate ROIs to obtain a pelletness score representing a probability that the object corresponding to the candidate ROI belongs to a feed pellet class; determining whether to retain the candidate ROI based on a degree of likelihood that the candidate ROI belongs to the feed pellet class; assigning each of the retained candidate ROIs to a trajectory; classifying the trajectories into a pellet trajectory or a non-pellet trajectory; and generating a pellet count in real time based on the pellet trajectory, representing a predetermined percentage of leftover feed pellets not consumed by the fish in the aquarium.

[0030] In at least one embodiment, the method further comprises transmitting the pellet count to a fish farm, where the pellet count is used by the fish feeder to generate a new amount of fish food to feed to the aquarium based at least in part on the pellet count.

[0031] In at least one embodiment, the method further comprises generating a control signal based on the pellet count and sending the control signal to a fish feeder at the fish farm to generate a new amount of fish feed to supply to the aquarium.

[0032] In at least one embodiment, the method further comprises determining gradients of the uneaten feed pellets over a given time interval and generating an alert if one of the gradients is greater than a gradient threshold.

[0033] In at least one embodiment, obtaining candidate ROIs in a given image frame comprises performing background subtraction on the given image frame to identify any objects likely to change position in subsequent image frames, performing contour detection on the identified objects to define boundaries that are potential ROIs for each identified object, and filtering the potential ROIs to obtain candidate ROIs.

[0034] In at least one embodiment, performing classification on the candidate ROIs comprises generating, for each candidate ROI, samples, where a given sample includes image data of the corresponding candidate ROI, pre-processing the samples, and applying an ROI classifier to each pre-processed sample to generate a probability distribution having probabilities that the sample belongs to different classes.

[0035] In at least one embodiment, the classes include a pellet class, a non-pellet class, and an ambiguous class.

[0036] In at least one embodiment, the pelletness score represents the probability that a sample belongs to the pellet class.

[0037] In at least one embodiment, the method comprises pre-processing the samples by resizing the samples to have the same matrix size and performing mean-center normalization.

[0038] In at least one embodiment, assigning each of the candidate ROIs in an image frame to one of the trajectories comprises: (a) assigning the candidate ROI to another new trajectory if there is no active trajectory; (b) assigning the candidate ROI to the closest active trajectory if a distance between a position of the candidate ROI and a position of the last ROI in the closest active trajectory is less than a distance threshold; or (c) assigning the candidate ROI to the new active trajectory if a distance between a position of the candidate ROI and a position of the last ROI in the closest active trajectory is greater than a distance threshold if there is an active trajectory.

[0039] In at least one embodiment, classifying the trajectories into pellet trajectories or non-pellet trajectories comprises identifying terminated trajectories including trajectories that have not been updated for one of the images in a given period of time, determining at least one trajectory measurement for each of the terminated trajectories, and determining which of the terminated trajectories belong to a pellet trajectory class based on the at least one trajectory measurement.

[0040] In at least one embodiment, determining which terminated trajectories belong to a pellet trajectory class comprises comparing at least one trajectory measurement to a threshold or providing the at least one trajectory measurement to a trajectory classifier.

[0041] In at least one embodiment, the at least one trajectory measurement includes at least one measure of the length of the completed trajectory, the distribution of pelletness scores of the ROI that comprises the completed trajectory, and / or the statistical distribution of one or more filter values.

[0042] In at least one embodiment, the method comprises mounting an object enhancer adjacent to the outlet of the discharge pipe opposite the feed camera and within the field of view of the feed camera to provide a background for the feed camera so that objects in a captured image have a definable surroundings, and securing the feed camera to the object enhancer such that a video feed captured by the feed camera comprises an image frame that is an image of the object enhancer and includes an object that crosses the field of view of the feed camera at the time the image is captured.

[0043] In at least one embodiment, the method comprises using a high density polyethylene (HDPE) sheet in the object enhancer to provide a background that provides increased contrast for objects passing between the feed camera and the object enhancer.

[0044] In another broad aspect of the teachings herein, there is provided a system for feed monitoring in a land-based fish farm having an aquarium containing fish, the system comprising: a feed camera attached to a discharge pipe coupled to the aquarium, the feed camera configured to capture a video feed comprising images of objects comprising pellets and non-pellet objects crossing a field of view of the feed camera at the time each image is captured; and a computing device comprising at least one processor and a non-transitory computer readable medium having instructions stored thereon that, when executed, cause the at least one processor to perform any of the methods described in accordance with the teachings herein.

[0045] Other features and advantages of the present application will become apparent from the following detailed description taken in conjunction with the accompanying drawings, but it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the present application, are given by way of illustration only, since various changes and modifications within the spirit and scope of the present application will become apparent to those skilled in the art from this detailed description.

[0046] For a better understanding of the various embodiments described herein, and to clearly show how these may be put into practice, reference is made to the accompanying drawings, which show, by way of example, at least one exemplary embodiment and are described below, and which are not intended to limit the scope of the teachings described herein. [Brief description of the drawings]

[0047] [Figure 1] FIG. 1 shows a schematic diagram of an exemplary embodiment of a system for AI-based feeding in fish farms. [Figure 2A] 1 shows a schematic diagram of an exemplary embodiment of an object enhancer. [Figure 2B] 2B shows a schematic diagram of an exemplary image for the object enhancer of FIG. 2A. [Diagram 3] 2 shows schematic diagrams of example screenshots of an API for use with the system of FIG. 1; [Figure 4] 4A and 4B show perspective and cross-sectional views of an exemplary embodiment of a bracket arrangement (e.g., a mounting assembly) on a portion of an outlet pipe where the bracket arrangement is coupled to the background enhancer of FIG. 2A. [Diagram 5] FIG. 2 shows a schematic diagram of the process flow of the system of FIG. 1. [Figure 6] 6A-6F show a flow chart of an exemplary embodiment of a method used by the system of FIG. 1 to automatically determine leftover fish feed (i.e. leftover feed pellets) by the fish. [Figure 7] 7A and 7B show perspective and cross-sectional views of another exemplary embodiment of an expandable and securable bracket arrangement (e.g., a mounting assembly) on a portion of an outlet pipe where the bracket arrangement is coupled to the background enhancer of FIG. 2A. [Figure 7C] 2B shows a perspective view of an exemplary embodiment of a bracket configuration for use with the background enhancer of FIG. 2A. [Figure 7D] 7D shows a front perspective view of the bracket configuration of FIG. 7C incorporating the background enhancer of FIG. 2A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] Additional aspects and features of the example embodiments described herein will become apparent from the following description, taken in conjunction with the accompanying drawings.

[0049] Various embodiments according to the teachings herein are described below to provide an example of at least one embodiment of the claimed subject matter. Any of the embodiments described herein are not intended to limit the claimed subject matter. The claimed subject matter is not limited to all of the features of any one of the devices, systems, or methods described below, or to devices, systems, or methods having features common to more than one or all of the devices, systems, or methods described herein. There may be devices, systems, or methods described herein that are not embodiments of the claimed subject matter. Any subject matter described herein but not claimed in this document may be the subject matter of another protective document, such as, for example, a continuing patent application, and the applicant, inventor, or owner does not intend to abandon, disclaim, or make available to the public any such subject matter by the disclosure of this document.

[0050] It will be understood that for simplicity and clarity of illustration, reference numerals may be repeated among the figures where appropriate to indicate corresponding or similar elements. Additionally, numerous specific details are described to provide a thorough understanding of the embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, this description should not be considered as limiting the scope of the embodiments described herein.

[0051] It should also be noted that the terms "coupled" or "couple" as used herein can have several different meanings depending on the context in which the terms are used. For example, the terms coupled or couple can have a mechanical meaning or an electrical meaning. For example, as used herein, the terms coupled or couple can indicate that two elements or devices can be directly coupled to each other, or that they are coupled to each other through one or more intermediate elements or devices, electrical signals, electrical connections, or mechanical elements, depending on the particular context.

[0052] It should also be noted that, as used herein, the word "and / or" is intended to represent an inclusive or. That is, "X and / or Y" is intended to mean, for example, X or Y or both. As a further example, "X, Y, and / or Z" is intended to mean X or Y or Z or any combination thereof.

[0053] It should be noted that terms of degree, such as "substantially," "about," and "generally," used herein, refer to a reasonable amount of deviation from the modified term such that the end result is not materially altered. These terms of degree may be interpreted as including deviations from the modified term, such as, for example, 1%, 2%, 5%, or 10%, for example, if such deviation does not negate the meaning of the term it modifies.

[0054] Additionally, the recitation herein of numerical ranges by endpoints includes all numbers and fractions subsumed within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It is also to be understood that all numerical values ​​and fractions thereof are presumed to be modified by the term "about," which refers to a variance of the referenced numerical value by up to a certain amount, such as, for example, 1%, 2%, 5%, or 10%, unless the end result would be significantly different.

[0055] It should be noted that use of the term "window" in connection with describing the operation of any system or method described herein is intended to be understood as describing a user interface, such as, for example, a graphical user interface (GUI), for performing initialization, calibration, or other user operations.

[0056] Exemplary embodiments of devices, systems, or methods described in accordance with the teachings herein are generally implemented as a combination of hardware and software. For example, the embodiments described herein may be implemented, at least in part, using one or more computer programs running on one or more programmable devices that include at least one processing element and at least one storage element (i.e., at least one volatile memory element and at least one non-volatile memory element). The hardware may include input devices including at least one of a touch screen, a keyboard, a mouse, a button, a key, a slider, and the like, and one or more of a display, a printer, one or more sensors, and the like, depending on the hardware implementation.

[0057] It should also be noted that some elements used to implement at least some of the embodiments described herein may be implemented via software written in a high level procedural language, such as object oriented programming. ++ , C#, JavaScript, Python, or any other suitable programming language and may comprise modules or classes as known to those skilled in the art of object oriented programming. Alternatively or additionally, some of these elements that are implemented via software may be written in assembly language, machine language, or firmware, as appropriate. In either case, the language may be a compiled or interpreted language.

[0058] At least some of these software programs may be stored on a computer readable medium, such as, but not limited to, a ROM, a magnetic disk, an optical disk, a USB key, etc., readable by a device having a processor, an operating system, and associated hardware and software necessary to perform at least one of the functions of the embodiments described herein. The software program code, when read by the device, configures the device to operate in a new specific predefined manner (e.g., as an application specific computer) to perform at least one of the methods described herein.

[0059] At least some of the programs associated with the device, system, and method embodiments described herein may be distributed in a computer program product comprising a computer-readable medium carrying computer usable instructions, such as, for example, program code, for one or more processing units. The medium may be provided in a variety of forms, including non-transitory forms, such as, for example, but not limited to, one or more diskettes, compact discs, tapes, chips, and magnetic and electronic storage. In alternative embodiments, the medium may be transitory in nature, such as, for example, but not limited to, wireline transmissions, satellite transmissions, Internet transmissions (e.g., downloads), media, digital, and analog signals. The computer usable instructions may be in a variety of formats, including compiled and non-compiled code.

[0060] In accordance with the teachings herein, various embodiments of systems and methods for AI-based feeding in fish farms, and computer products for use therewith, are provided.

[0061] Referring initially to FIG. 1, a schematic diagram of an exemplary embodiment of a feed counting system 100 for AI-based feeding in fish farms is shown. The feed counting system 100 includes a fish tank 110, a discharge pipe 112, a feed camera 114 fixed to the discharge pipe 112, an object enhancer 116, an Ethernet cable 118, an Ethernet switch 120, and a computer 140. The discharge pipe 112 may also be referred to as an exit pipe. The object enhancer 116 may also be referred to as a profile enhancer or a background enhancer. It should be noted that in alternative embodiments, some elements may be implemented using other elements, such as the Ethernet cable 118 and the Ethernet switch 120, which may be implemented using other networking elements.

[0062] The feed camera 114 (which may be a single IP camera) and the object enhancer 116 are installed in the discharge pipe 112 generally opposite each other. As camera technology continues to develop over the years, the technical specifications of the feed camera 114 may evolve over time. However, a particular specification of the feed camera 114 that may be considered is the resolution and exposure. For example, a resolution of 576×704 and an exposure of 1 / 250 may be suitable for use in the system 100. This allows for the collection of high quality data, as the object enhancer 116 provides a background to the field of view of the feed camera 114, such that an object passing between the feed camera 114 and the object enhancer 116 and captured in an image acquired by the feed camera 114 has a definable boundary in at least one of the acquired images in the video feed.

[0063] The shape of the object enhancer 116 may be generally circular with a groove to receive a fastener for fixing it in place in the discharge pipe 112 (e.g., as shown in FIG. 2A). The feed counting system 100 utilizes a pellet counting algorithm (or "pellet tracker") that tracks the amount of waste feed flowing out of the aquarium 110 through the discharge pipe 112 in real time by distinguishing between pellets 210, feces 220, and other elements (e.g., as shown in FIG. 2B). The feed counting system 100 can be directly tied to the API 310 of an existing feeding system to automatically update the amount of feed administered by the fish feeder 320 used by the feed counting system 100 according to the appetite of the fish in the aquarium 110 (e.g., as shown in FIG. 3) based on the uneaten feed (e.g., the number of uneaten feed pellets) determined by the pellet tracker. The fish feeder 320 is a system utilized in fish farms to provide fish feed (i.e., pellets) to fish at specific time intervals, as known to those skilled in the art.

[0064] The computer 140, which may be referred to as a computing device, may be implemented as a single computing device (e.g., a desktop, laptop, or notepad) and includes a processor unit 144 (which may be referred to as having one or more processors, or simply referred to as a processor for ease of illustration), a display 146, an interface unit 148, input / output (I / O) hardware 150, a communication unit 152, a power supply unit 154, and a memory unit 156 (also referred to as a "data store"). The memory unit 156 may be a non-transitory computer-readable medium on which computer instructions for use by the processor unit 144 may be stored. In other embodiments, the computer 140 may have more or fewer components, but generally function in a similar manner. For example, the computer 140 may be implemented with multiple computing devices and / or processor units 144. For example, the computer 140 may be implemented to function as a server or a server cluster.

[0065] Processor unit 144 may include one processor that controls the operation of system 100 and may provide sufficient processing power depending on the configuration and operational requirements of system 100. For example, processor unit 144 may possibly include a high performance processor or a GPU. Alternatively, there may be multiple processors used by processor unit 144, which may function in parallel to perform certain functions.

[0066] The display 146 may be, for example but not limited to, a computer monitor or LCD display for a tablet device or a desktop computer.

[0067] The interface unit 148 may be any interface that allows the processor unit 144 to communicate with other devices in the system 100. In some embodiments, the interface unit 148 may include at least one of a serial or parallel bus and a corresponding port, such as, for example, a parallel port, a serial port, a USB port, and / or a network port. For example, a network port may be used to enable the processor unit 144 to communicate over the Internet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a wireless local area network (WLAN), a virtual private network (VPN), or a peer-to-peer network, either directly or through a modem, router, switch, hub, or other routing or conversion device.

[0068] I / O hardware 150 may include, for example, but is not limited to, a microphone, a speaker, a keyboard, a mouse, a touchpad, a display device, and a printer.

[0069] Power supply unit 154 may include one or more power supplies (not shown) connected to provide power to the various components of system 100, as is known to those skilled in the art.

[0070] Communications unit 152 includes various communications hardware to enable processor unit 144 to communicate with other devices. For example, communications unit 152 includes at least one of a network adapter, such as an Ethernet or 802.11x adapter, a Bluetooth radio or other short-range communications device, or a wireless transceiver for wireless communications conforming to the CDMA, GSM, or GPRS protocols using standards such as IEEE 802.11a, 802.11b, 802.11g, or 802.11n.

[0071] The memory unit 156 stores program instructions for the operating system 158, the pellet tracker 159, the region of interest (ROI) proposal module 160, the ROI classification module 162, the ROI tracking module 164, the trajectory classification module 166, and the data file 168. When any of the program instructions for the pellet tracker 159 or any of the modules 160-166 are executed by at least one processor of the processor unit 144 or a processor of another computing device, the at least one processor is configured to perform a particular function according to the teachings of the present teachings. It should be noted that in alternative embodiments, the pellet tracker 159 and the modules 160-166 may be organized in other ways using different software programming structures so long as the same functionality is provided. The operating system 158 may select which physical processor is used to execute particular modules and other programs. For example, the operating system 158 may switch processes to run on different portions of the physical hardware used, for example, using different cores in a processor or using different processors on a multiprocessor server.

[0072] The pellet tracker 159 includes program instructions for performing pellet counting in real time and generates feed data, such as the amount of uneaten feed pellets and / or any gradient fish feed data regarding the spike of uneaten feed pellets, which is provided to the API 310 for generating control signals to the fish feeder 320 to adjust the amount of fish feed provided to the fish in the aquarium 110. The pellet tracker 159 calls the ROI proposal module 160, the ROI classification module 162, the ROI tracking module 164, and the trajectory classification module 166 to take specific actions during pellet tracking. The operation of the pellet tracker 159 is generally described with respect to FIG. 6A.

[0073] Alternatively, in at least one embodiment, the various modules may function as standalone programs that communicate via message passing, e.g., in an event-based fashion. For example, the ROI proposal module 160 may be communicatively coupled to a video feed, and as new image frames are acquired, the ROI proposal module 160 processes them. As ROIs for image frames are prepared for further processing, the ROI classification module 162 waits to accept them. Similarly, the ROI classification module 162 generates events for the ROI tracker module 164, notifying it that there is data to process. This allows the software architecture to be scalable and distributed across multiple processors and / or computing devices in a secure and scalable manner.

[0074] The data file 168 may be used to store operational parameters, such as particular classifiers and other variables, used by the pellet tracker 159 and modules 160-166. The data file 168 may also be used to store the history of leftover feed pellets as determined by the feed tracker, as well as other data, such as identified objects and trajectories, that may be used for quality control and / or further training of algorithms used by the modules 160-166. High-level overview

[0075] At a high level, the pellet counting algorithm function can be decomposed into four distinct steps: region of interest (ROI) proposal, ROI classification, ROI tracking, and trajectory classification.

[0076] The ROI proposal module 160 reads video frames and finds segments in each frame that roughly represent discrete objects. The ROI proposal module 160 uses computer vision techniques in an initial round of filtering that excludes ROIs that are clearly not pellets to generate a set of proposed ROIs. The output of the ROI proposal module 160 includes the proposed ROIs along with any necessary metadata that may be provided to the ROI classification module 162. The operation of the ROI proposal module 160 is generally described with respect to FIG. 6B.

[0077] The ROI classification module 162 is responsible for assigning to each ROI determined by the ROI proposal module 160 a probability score of "pelletness" that represents the probability that the ROI actually represents an object that is a feed pellet. To assign a pelletness score to each ROI, the ROI classification module 162 may use deep learning (DL) based techniques, such as, but not limited to, convolutional neural networks (CNNs) operating on the raw pixels of the ROI, or more "classical" machine learning techniques (e.g., tree-based methods such as extreme gradient boosted trees (XGBs), support vector machines (SVMs), etc.) that operate using specific features extracted in the ROI proposal module 160 (e.g., pixel color distribution, edges discovered using Canny or Sobol edge detection, etc.). The ROI classification module 162 may also be more generally referred to as a "classifier." The output of the ROI classification module 162 includes a pelletness score for each ROI, as well as any necessary metadata that may be provided to the ROI tracking module. The operation of the ROI classification module 162 is generally described with respect to FIG. 6C.

[0078] The ROI tracking module 164 uses a custom designed tracking algorithm to determine and update pellet trajectories by determining whether an ROI with an associated pelletness score can be (1) added to a new trajectory, (2) added to an existing trajectory, or (3) ignored. Each trajectory includes the Cartesian coordinates of each ROI that comprises the trajectory and the corresponding pelletness score. After the ROI tracking module 164 determines what to do for each ROI that appears in a given frame, it terminates any trajectories that have not been updated with a new ROI that is added to one or more terminated trajectories. The output of the ROI tracking module 164 includes the terminated trajectories, which can be sent to the trajectory classification module 166. The operation of the ROI tracking module 164 is generally described with respect to FIGS. 6D-6E.

[0079] The trajectory classification module 166 makes the final decision whether the terminated trajectory counts as a pellet or a non-pellet object. To do so, the trajectory classification module 166 may use data such as the length of the trajectory, the distribution of pelletness scores of the objects that make up the trajectory, and the distribution of other ROI metadata determined by the ROI proposal module 160. The trajectory classification module 166 may process the above-mentioned data using a probabilistic machine learning model or simply using carefully selected thresholds and conditional rules to determine whether the terminated trajectory corresponds to a feed pellet or to another object. The operation of the trajectory classification module 166 is generally described with respect to FIG. 6F. Physical setup for data collection

[0080] In order for the pellet counting algorithm to function effectively, improved efficiency and accuracy are achieved through a specific hardware setup, as described below.

[0081] The object enhancer 116 may be made of any material that creates a background for the feed camera 114. For example, but not limited to, the object enhancer 116 may be a durable, translucent plastic made using high density polyethylene sheeting (HDPE) or other materials. The object enhancer 116 creates a high contrast background so that any objects passing between the feed camera 114 and the object enhancer 116 are more easily identifiable by algorithms applied to image frames from the video feed. Preferably, the object enhancer has a consistent color and is translucent. For example, but not limited to, the color may be white or light (e.g., to provide a high contrast background).

[0082] FIG 4A shows a perspective view of an exemplary embodiment of a hardware setup 400 used in the discharge pipe 112. FIG 4B shows a cross-sectional view of the exemplary embodiment shown in FIG 4A. The feed camera 114 may be coupled to the underside of the object enhancer 116 such that the field of view of the feed camera 114 may capture all or most of the object enhancer 116 and such that there is a certain distance between the feed camera 114 and the underside of the object enhancer 116. For example, the bracket 408 may have an upper end attached to the underside of the object enhancer 116 and a middle part coupled to the upper end via legs, and the feed camera 114 may be attached to the middle part and have its field of view directed toward the object enhancer 116. A second bracket, such as, for example, a U-shaped bracket having a strut 410p, may be attached to the discharge pipe 112 in a manner that limits the ability of the object enhancer 116 and the feed camera 114 to rotate, but allows the object enhancer 116 and the feed camera 114 to move up and down as water pressure changes in the discharge pipe 112 cause the water flow 460 to exit the discharge pipe 112. The bracket 408 and / or the second bracket 410 may be coupled to the discharge pipe 112 in other manners such that the bracket 408 and / or the second bracket 410 is adjacent to the discharge pipe 112 and maintains a position relative to the discharge pipe 112 to allow the feed camera 114 and the object enhancer 116 to remain at a constant distance or to vary their distance within a predetermined range. Alternatively or additionally, the feed camera 114 may be coupled to the bracket 408 or the second bracket 410. Alternatively or additionally, the feed camera 114 may be coupled to the discharge pipe 112. The feed camera 114 may have a wire 420 for connecting to a network connection (e.g., an Ethernet cable 118). Alternatively, the feed camera 114 may have a wireless connection to an Ethernet switch 120.

[0083] For example, the bracket 410 may be attached to the outlet port or opening of the outlet pipe 112 such that the object enhancer 116 is located above the outlet port of the outlet pipe 112 and may move upward to provide a gap between the bottom surface of the object enhancer 116 and the top surface of the outlet port to allow water flow 460 to exit the outlet pipe 112 if water pressure increases within the outlet pipe 112. To accomplish this, the bracket 410 includes a sliding mechanism in the form of a post 410p slidably engaged by grooves 116g present in two portions of the edge of the object enhancer 116. If water pressure decreases, the object enhancer 116 may move downward. If water pressure increases, the object enhancer 116 may move upward. As a result, a configuration can be created in which the feed camera 114 is fixed to the outlet pipe 112 and the object enhancer 116 is slidably mounted adjacent to the outlet pipe 112 such that the object enhancer 116 can move up and down as different amounts of water flow exit the outlet pipe 112.

[0084] Although not shown, a retaining element may be used to prevent the object enhancer 116 from dislodging from the post 410p of the bracket 410 and lifting out of the exit port of the outlet conduit 112. For example, the end of the post 410p may have a bulbous end or may be flared so as to engage the top surface of the object enhancer 116 and prevent the object enhancer 116 from dislodging from its mounting (e.g., the post 410p of the bracket 410) at the exit port of the outlet conduit 112 if the water flow 460 is strong enough to lift the object enhancer 116 up to the top of the post 410p.

[0085] Optionally, in at least one embodiment, a light source 450 may be mounted to generate light directed toward the object enhancer 116. In such case, the light source 450 may be secured to the bracket 410.

[0086] Bracket 408, bracket 410, and post 410p may be collectively referred to as a bracket arrangement or mounting assembly.

[0087] It should be noted that in at least one embodiment, the bracket 410 includes lower horizontal members 410m1 and 410m2 that are adjacent to one another and movable relative to one another to allow the bracket configuration to slidably expand or contract in width so that the bracket configuration can accommodate outlet pipes 112 having different diameters. Thus, the lower horizontal members 410m1 and 410m2 include aligned channels that can receive fasteners to secure the two horizontal members 410m1 and 410m2 to one another when the two horizontal members 410m1 and 410m2 are positioned such that the upper lateral tabs of the bracket 410, on which the struts 410p are positioned, overlap the upper edge of the side wall of the outlet pipe 112.

[0088] 5 shows a schematic diagram of an exemplary embodiment of the system 100 in an operating environment. The aquarium 110 has fish 510 therein that eat pellets 210. The fish 510 also excrete feces 220. The uneaten feed pellets 210 and feces 220 are discharged from the aquarium 110 through the discharge pipe 112. These discrete objects (uneaten feed pellets 210 and feces 220) are moved toward the object enhancer 116 by water pressure and water flow 460. The object enhancer 116 floats on the water surface and movably engages with the post 410p of the tube bracket 410 so as not to block the water flow 460. The feed camera 114 is aimed at the object enhancer 116, which is translucent or transparent, and a light source 450 illuminates the object enhancer 116. In at least one implementation, the light source 450 illuminates the object enhancer 116 only when the ambient light in the operating environment is insufficient to allow objects (e.g., uneaten feed pellets 210, feces 220, and other elements) to be captured in the image frame of the video feed 530 acquired by the feed camera 114. The feed camera 114 provides the video feed 530 to the computer 140 via a communication link, such as, for example, an Ethernet cable 118, or wirelessly in at least one embodiment in which the feed camera 114 includes a transceiver (not shown). The computer 140 operates a pellet tracker 159 that counts uneaten feed pellets and provides a data output, such as, for example, pellet count, to the fish feeder 320. In at least one embodiment, the computer 140 may provide the pellet count and optionally any excess gradient in uneaten feed pellets to the fish farm operator for future use or tracking. The computer 140 may create this output using various modules described below.

[0089] FIG. 6A shows a flow chart of an example embodiment of a pellet counting algorithm 600 that may be implemented by the pellet tracker 159 for use by the system 100.

[0090] At 610, computer 140 obtains the resulting image frames from video feed 530 acquired by feed camera 114. For example, computer 140 acquires the image frames in the order acquired by feed camera 114, for example, at a predetermined frame rate. In at least one implementation, feed camera 114 uses a frame rate of 25 frames per second; if the frame rate is too slow, objects may move too far between frames to allow accurate object tracking. The image frames may be stored in data store 156. It should be understood that each image frame includes data representing what was in the field of view of feed camera 114 when the frame was acquired.

[0091] At 620, the computer 140 pre-processes the image frames using the ROI proposal module 160 to obtain a proposed ROI and any associated metadata (such as a unique identifier for the feed camera 114 from which they originate). The proposed ROI and metadata are provided to the ROI classification module 162 via a messaging tool, such as, but not limited to, Redis or Apache Kafka. The messaging tool used may facilitate data transfer between different modules, for example, when each module runs on a different processor or different computing device in a cluster. The operation of the ROI proposal module 160 is described in more detail with respect to FIG. 6B.

[0092] At 630, computer 140 performs ROI classification using ROI classification module 162 to obtain a pelletness score for each ROI along with any associated metadata. The metadata may be, for example, a unique ID that informs one or more of the downstream modules which camera the ROI came from, and a timestamp that uniquely identifies each frame. The pelletness scores and metadata are provided to tracking module 164, again via messaging tools. The operation of ROI classification module 162 is described in more detail with respect to FIG. 6C.

[0093] At 640, computer 140 performs ROI tracking on the ROI, the associated pelletness score, and any applicable metadata using ROI tracking module 164. ROI tracking module 164 generates new trajectories, updates trajectories, and / or determines terminated trajectories (trajectories that cannot be updated). The output of ROI trajectory module 164 is the terminated trajectory. The operation of ROI tracking module 164 is described in more detail with respect to Figures 6D and 6E.

[0094] At 650, computer 140 performs trajectory classification on the terminated trajectory using trajectory classification module 166. The output of trajectory classification module 166 is whether the terminated trajectory is counted as a pellet or a non-pellet object. The operation of trajectory classification module 166 is described in more detail with respect to FIG. 6F.

[0095] At 660, the results of the trajectory classification are provided to the fish farm and processed to determine data about which decisions can be made at the fish farm. For example, the results of the trajectory classification can be processed at 660 such that the amount of uneaten feed pellets or whether there is a large gradient in uneaten feed pellets over a particular time interval, such as, for example, 1 minute, 5 minutes, 10 minutes, 15 minutes, or more, is provided directly to the API 310 or the feeder 320, which acts to control the amount of fish feed provided by the feeder 320 to the fish in the aquarium 110. In at least one implementation, the time interval can be pre-programmed. Alternatively, in at least one implementation, the time interval can be modified during use based on input provided by an end user, for example, via the API 310.

[0096] In at least one embodiment, the pellet counting algorithm 600 determines the number of waste pellets passing through the field of view of the feed camera 114 as a predetermined percentage (e.g., 10%, 15%, 20%, or another percentage) of the total number of waste pellets exiting through the discharge pipe 112. This predetermined percentage may be established in various ways, such as, for example, by selecting a field of view and / or size of the object enhancer 116 that results in a known ratio of objects passing through the field of view of the feed camera 114 compared to the total number of objects exiting through the discharge pipe 112, where the objects counted and the total number of objects are uneaten feed pellets. Alternatively or additionally, this predetermined percentage may be verified by measurement, such as, for example, by measuring / counting the objects passing through the field of view of the feed camera 114, using a net / container to collect all objects exiting through the discharge pipe 112 and counting the objects in the net / container. This predetermined percentage provides a scale factor (e.g., 10 if the predetermined percentage is 10%, or 5 if the predetermined percentage is 20%, or other scale factor depending on the actual physical setup of the feed camera, object enhancer, and discharge pipe) that can then be used to extrapolate the total number of waste pellets exiting through the discharge pipe 112.

[0097] Alternatively, in at least one implementation, computer 140 provides the results of the trajectory classification to a backend service that may be part of system 100 that manages a database, displays the results to an end user, and integrates with API 310 using a framework of rules that may be updated by the end user. For example, such an update rule may be "if the number of pellets counted in the last 10 minutes was below a certain number, increase the feeding rate by 5%, and if it is above a certain number, decrease the feeding rate by 5%." In some implementations, the rate at which the feeding rate is increased or decreased may be variable, depending on how far the number of pellets counted is from the ideal number of discarded pellets (i.e., the set point). ROI Recommendation Module Details

[0098] 6B shows a flow chart of an exemplary embodiment of a method for performing pre-processing 620 performed by ROI proposal module 160. ROI proposal module 160 operates in an environment where only pixels that change significantly between two consecutive image frames are potentially representative of discarded feed pellets or other moving objects, as determined by object enhancer 116. This environmental characteristic is used to identify regions of each image frame that warrant further consideration (e.g., regions of interest ROIs).

[0099] At 622, the computer 140 performs background subtraction. The technique used to perform background subtraction is one that is capable of identifying objects that may move between subsequent frames. For example, upon loading a new image frame, the ROI proposal module 160 may generate a binary pixel mask for the new image frame using a background subtraction algorithm, such as, but not limited to, a K-nearest neighbor (KNN) based algorithm. In this mask, pixels with a value of 0 are considered to be part of the non-moving background, and pixels with a value of 1 are part of the moving foreground, and such pixels may be associated with a moving object.

[0100] At 624, the computer 140 performs contour detection to determine potential ROIs. For example, the ROI proposal module 160 uses a contour detection algorithm to extract the boundaries of each "island" of foreground pixels in the binary pixel mask. The contour detection algorithm may be implemented using a variety of suitable techniques, including, for example, but not limited to, pixel-following, vertex-following, or run-data-based following techniques. An example of a pixel-following contour detection algorithm that may be used is the findContours method provided by the OpenCV library. By finding the minimum and maximum points of each contour in both the x and y directions, the ROI proposal module 160 may define bounding boxes around each foreground object. The contents of these bounding boxes in the original image are referred to as the ROI.

[0101] In reality, in a given image, there are many patches of varying pixels that do not correspond to feed pellets. Such pixel patches may represent other objects such as excrement, air bubbles, or moving devices, but often originate from visual artifacts such as water / air interface shimmer. To prevent wasting computational resources, the ROI proposal module 160 may be configured to remove non-pellet ROIs before sending them for further processing by performing filtering at 626.

[0102] At 626, the computer 140 performs ROI filtering 626 to identify and remove any ROIs that do not correspond to feed pellets. For example, (1) one or more computer vision based filters may be used to extract features from each ROI, and (2) these features may then be compared to empirically determined thresholds to determine which ROIs may be omitted based on the features described above. Examples of filters that may be used include, but are not limited to, a minimum outline size requirement to filter ROIs that correspond to objects and artifacts that are too small to be feed pellets. Alternatively, another type of filter may be based on shape requirements, such as circularity or aspect ratio.

[0103] Alternatively, more complex sets of specially designed filters are effective for removing certain types of undesired objects, e.g., air bubbles and water shimmer artifacts. One approach is to compare the intensity of pixels within the contour of the ROI with pixels outside the contour. Pixels within the contour defining the feed pellets are expected to be darker than the surrounding pixels. In contrast, contours defining water artifacts and air bubbles generally do not meet this assumption.

[0104] Yet another ROI filtering method that may be used is based on a comparative measurement of a contour and its counterpart "enhanced contour." In contrast to the ROI contour generated from the foreground mask (e.g., the output of background subtraction), in at least one embodiment, the ROI proposal module 160 may generate the "enhanced contour" by thresholding the original image to obtain a second estimate of where the object boundary is located. This thresholding may use, for example, the object's characteristic of being dark and the object enhancer's characteristic of being light to find the object boundary. The shapes of the two different types of contours obtained in the first and second estimates are similar for food pellets, but are significantly different for air bubbles, water artifacts, and other objects with irregular edges and extrusions (e.g., feces, etc.). Thus, shape measurements may be obtained for object contours that correspond to each other (e.g., representing the same potential object) from the first and second estimates, and if the difference in shape measurements is less than the shape measurement threshold level, those ROIs are likely to be for the same object (e.g., food pellets) and may be retained, while other ROIs with shape measurement differences greater than the shape measurement threshold may be removed.

[0105] Although these filtering methods may not be sufficient to identify pellet ROIs with high accuracy and do not provide a "pelletness" probability, they do serve to remove the majority of non-feed pellet ROIs. Removing these non-feed pellet ROIs at this stage significantly improves the overall computational efficiency downstream of the pellet counting algorithm.

[0106] In at least one embodiment, multiple filters may be used in a serial fashion to remove non-feed pellet ROIs. For example, a first filter stage may be based on a first feature measured from the candidate ROI, and each unfiltered ROI is provided to a second filter stage based on a second measure or enhanced contour filtering, and so on, possibly for additional filter stages.

[0107] Also, in at least one embodiment, the ROI filter can be used to identify if there is anything abnormal with the physical setup in the outlet pipe (e.g., the hardware setup of Figures 4A and 4B) and notify an end user if, for example, there is an abnormal amount of air bubbles based on the use of a filter capable of identifying air bubbles.

[0108] At 628, computer 140 determines a set of candidate ROIs to be processed in a given image frame based on the ROIs that were not filtered at 626. Computer 140 also generates metadata, such as, for example, a unique identifier for feed camera 114 and a timestamp that uniquely identifies each of the image frames captured by feed camera 114. ROI Classification Module Details

[0109] 6C shows a flow chart of an example embodiment of a ROI classification method 630 that may be performed by the ROI classification module 162. The main purpose of the ROI classification module 162 is to assign to each ROI a probability that the ROI is a pellet (a "pelletness" score). Although there are several ways in which the method 630 may be implemented, it is preferred to use a convolutional neural network (CNN) to obtain the pelletness score.

[0110] Many classifiers based on CNN models have achieved good results on complex datasets. However, due to the nature of the binary pellet / non-pellet classification task, it is preferable to use a relatively low-complexity CNN model to prevent overfitting and an unnecessarily large computational footprint. One possible implementation of the ROI classifier is to use the MobileNet v2 model with an alpha parameter set to 0.25 as the feature extraction network, followed by a dense connection layer with 64 hidden units with a ReLU activation function, and a final dense connection layer with three output nodes and a softmax activation function. However, it should be noted that other classifiers may be used in alternative implementations.

[0111] Prior to inputting samples (e.g., each sample corresponds to a given ROI from the current image frame being processed and contains image data within the ROI from that image frame) into the ROI classifier for training or inference, pre-processing is performed on the ROIs such that they are reshaped to a size of 64x64 pixels and also undergo mean-centered normalization. These 64x64 RGB image input datasets are the only inputs to the ROI classifier. During training, random augmentation in the form of rotations and horizontal / vertical flips is used. Since the neural network is only intended to recognize complete pellets, augmentation that may prevent the entire pellet from being shown (e.g., random cropping or shearing) is not used. No pre-trained weights are used (i.e., there is no transfer learning).

[0112] A common problem with neural networks is poor calibration, meaning a tendency to only make confident predictions. This is an undesirable quality in that when a given CNN model makes an incorrect decision, it is accompanied by a high degree of confidence. In the distribution of images in the production environment from which the dataset is acquired for processing by the ROI classifier, there are many objects for which even a human labeler cannot be confident in deciding whether the sample is a pellet or a non-pellet object. In such cases, it is desirable for the pellet classification model to generate probabilities that reflect this uncertainty (i.e., probabilities closer to 0.5 than 0 or 1).

[0113] To mitigate the problem of poor model calibration, it is helpful to build a training dataset with objects in three classes: pellets, non-pellets, and ambiguous. Human labelers were instructed to label ROIs provided by the ROI proposal module 160 as one of these three classes and to use the ambiguous class whenever they were not completely confident in the pellet / non-pellet prediction.

[0114] One important difference between how human labelers and CNNs look at ROI data is that humans can use contextual information from surrounding (e.g., previous and next) image frames. If there is uncertainty about whether an ROI can be labeled as a pellet, humans can look at both the previous and next frames to help them make the decision. In such cases, if the ROI is clearly a pellet based on information from the surrounding frames (e.g., how it looked in previous frames, or how it moved between frames), but it is not clear what a pellet is just by looking at the object in each individual frame, the labelers were instructed to forget the contextual information and use an ambiguous label.

[0115] The addition of the ambiguity class enhanced the meaningfulness of the probability distribution output by the ROI classifier for each sample. Here, a sample is a single object, and the probability distribution is defined as the probability that the object is (a) a pellet, (b) a non-pellet, and (c) ambiguous, with the sum of these three probabilities equal to 1 (i.e., it is a softmax distribution). In some scenarios, a genuine pellet trajectory may have one ROI in the middle that slightly overlaps with a non-pellet object (e.g., an air bubble or part of excrement), which will result in the ROI being assigned a pelletness score of 0.0 and omitted by the tracker. By employing this method, the classification model may give the ROI a low score, but not so low that it is ignored by the tracker. Employing this technique also improves the ability of the trajectory classification module 166 to effectively separate pellet trajectories from non-pellet object trajectories.

[0116] To achieve sufficient performance, at least 5000 samples of each class can be used to train the ROI classification model. These samples can be obtained from as many scenarios as possible (e.g., different lighting conditions or outfall flow rates). When moving to a new environment, fine-tuning of the classification model may be required.

[0117] At 632, computer 140 obtains the current image frame and coordinates of the ROI being processed. Samples representing image data for each ROI are obtained. At 634, the samples are pre-processed, which may be done as described above by performing reshaping and mean-centered normalization so that each is represented by a matrix having the same size.

[0118] At 636, each pre-processed sample is applied to an ROI classifier that generates a probability distribution for each sample. The ROI classifier may be a CNN-based classifier as described above. It may be a discrete probability distribution, such as a softmax distribution. For example, for each sample, the probability distribution includes a probability score that sums to 1 and represents the likelihood that the potential object belongs to one of the classes used by the ROI classifier. The probability distribution for a given sample includes a first probability that the object corresponding to the sample is a pellet, a second probability that the object is a non-pellet, and a third probability that the object is ambiguous. For example, the CNN classifier may provide a probability distribution for a given sample as 0.8 for the pellet class, 0.15 for the ambiguous class, and 0.05 for the non-pellet class.

[0119] The highest probability for each sample is used to label the object ROI corresponding to the sample as either pellet, non-pellet, or ambiguous at 638. Data file 168 may be updated to keep track of the classification of each ROI processed by ROI classification module 164. ROI Tracking Module Details

[0120] 6D shows a flow chart of an example embodiment of an ROI tracking method 640 that may be performed by the ROI tracking module 164 based on an ROI classified as a pellet. Because the objective is to count pellets and pellets are present in multiple image frames of the video feed, the pellet tracker 159 may track each pellet that is identified to avoid counting a pellet multiple times across consecutive image frames.

[0121] Many open source implementations of tracking algorithms, such as OpenCV, are insufficient or ineffective. Typically, these algorithms work by initializing with an image patch (e.g., a ROI) and searching for a similar image patch in the next image (e.g., the next frame in a video). Because these methods operate on image patches, they are computationally intractable at the scale of modern land-based fish farms. Also, they do not provide the opportunity to use a pelletness score when deciding whether to add an ROI to an existing trajectory.

[0122] However, in accordance with the teachings herein, the pellet counting algorithm does not operate using image patches, but rather using a pelletness score and the coordinates of the centroid of each ROI. In general, the pelletness score is used to determine whether an ROI is worthy of tracking, and if so, whether to (1) start a new trajectory with the current ROI being processed, or (2) add the ROI to an existing trajectory.

[0123] At 642, the ROI tracking module 164 receives ROI data from a single image frame for a given ROI classified as a pellet (i.e., a "pellet ROI"). This ROI data may include the location of the ROI (e.g., coordinates of the ROI) and a "pelletness ROI score," which is the probability that the ROI corresponds to a pellet as determined by the ROI classification module 162. This ROI data may also include metadata, such as an identifier for the feed camera 114 and a timestamp representing the time when the image frame currently being processed was acquired.

[0124] At 644, the computer 140 determines whether the ROI score meets a threshold score, which is another test that the ROI is more likely to be a pellet than to belong to another class after the classification made by the ROI classifier. If this comparison is true (i.e., yes, the ROI score meets the threshold score), the ROI can be termed a candidate ROI and method 640 proceeds to 645.

[0125] If the comparison at 644 is false (i.e., no, the ROI score does not meet the threshold score), the method 640 proceeds to 646, at which point the object is discarded and not tracked further. The threshold score may be determined empirically.

[0126] At 645, the computer 140 determines which trajectory includes the candidate ROI, which may include comparing the position of the object corresponding to the ROI pellet with the trajectory currently being tracked. For example, if the distance between the candidate ROI being evaluated and the "tail" (e.g., the last ROI) of the existing trajectory closest to the candidate ROI being evaluated is greater than a predetermined distance threshold (e.g., in terms of Euclidean distance), a new trajectory with the current ROI may be started using the current ROI being evaluated as a starting point. Alternatively, if the distance is less than the distance threshold, the ROI tracking module 164 adds the current ROI being evaluated to the existing closest trajectory. The evaluation at 646 may be performed, for example, using the method 700 of FIG. 6E.

[0127] At 647, it is determined whether there are more ROIs to evaluate for the current image frame. If there are more ROIs to evaluate, the method 600 returns to 642.

[0128] If all ROIs have been processed for the current image, method 640 proceeds to 648 where ROI tracking module 164 determines if any trajectories that existed prior to the current image frame being processed have not been updated in any of the ROIs in the current image frame. Each of these trajectories is referred to as a "terminated trajectory." ROI tracking module 164 identifies any terminated trajectories, which may then be provided to trajectory classification module 166.

[0129] 6E shows a flow chart of an exemplary embodiment of a method 650 for assigning an ROI to an existing or new trajectory. The method 650 may be performed by the ROI tracking module 164 when executed by the computer 140. In this exemplary embodiment, the method 650 operates on all ROIs in the current image frame that have not been discarded based on the ROI pelletness score as described above with respect to FIG. 6D, and is implemented as a batch processing method that can process several image frames consecutively. In at least one implementation, the method 650 is executed by the computer 140 for each image frame, with the computer 140 receiving as input all ROIs found in a particular image frame.

[0130] At 652, computer 140 obtains candidate ROIs and pelletness scores for the current image frame. At 654, it is determined whether there are no candidate ROIs. If this determination is true, method 650 may proceed to 656, where the next image frame is loaded, after which method 650 returns to 652, where data for the candidate ROIs for the next image frame is obtained. Alternatively, if the determination at 654 is false, method 650 proceeds to 658.

[0131] At 658, it is determined whether there are no candidate ROIs and there is an active trajectory. If this determination is true, meaning there is no ROI to add to the active trajectory, then method 650 proceeds to 660. If the determination at 658 is false, then method 650 proceeds to 662.

[0132] At 660, the active trajectory may be terminated and subsequently identified as a terminated trajectory. An active trajectory is one to which an ROI was added in a previous image frame. At this point, the method 650 may proceed to 656 to process candidate ROIs in the next image frame, or the method 650 may end (not shown).

[0133] At 662, it is determined whether a candidate ROI exists and there is no active trajectory. If this determination is true, meaning there is no active trajectory to add the candidate ROI to and the candidate ROI can be added to a new trajectory, in which case method 650 proceeds to 664. If the determination at 662 is false, method 650 proceeds to 666.

[0134] At 664, a new active trajectory is created for each candidate ROI, and the candidate ROI is the starting node or object for the new trajectory. At this point, the method 650 may proceed to 666 to process the candidate ROIs in the next image frame, or the method 650 may end (not shown).

[0135] At 666, initially there is a situation where there are several (i.e., at least one) candidate ROI and at least one active trajectory. Method 650 repeats 666, 668, 670, and 672, and each time 666 is executed, it is determined whether there are any candidate ROIs that need to be evaluated. If this determination is true, method 650 proceeds to 668. If this determination is false, method 650 proceeds to 660.

[0136] At 668, the method 650 includes determining whether the candidate ROI belongs to the active trajectory or whether one or more new active trajectories should be created. This may be determined by locating a candidate ROI that matches or corresponds to the active trajectory based on a comparison of the distance from the candidate ROI to the active trajectory and a distance threshold. This may be done by determining a pairwise distance for the current candidate ROI's position (e.g., centroid) and its (e.g., Euclidean) distance from the end position of the active trajectory (e.g., the position of the most recently added ROI), and determining the minimum pairwise distance.

[0137] The method 650 proceeds to 670 where the minimum pair-wise distance between each candidate ROI and its corresponding closest active trajectory is compared to a distance threshold. If the pair-wise distance is less than the distance threshold, the method 650 proceeds to 672.

[0138] The candidate ROI is added to the closest active trajectory at 672. The candidate ROI and its closest active trajectory are then removed from consideration and the method 650 proceeds to 666 to determine if there are other candidate ROIs that need to be evaluated.

[0139] If the determination at 670 is false, method 650 proceeds to 664, at which point a new trajectory is created for the current candidate ROI. Method 650 then proceeds to 666, where it is determined whether there are more candidate ROIs to evaluate. Before method 650 proceeds to 666, computer 140 removes the last created trajectory and ROI from consideration in the next loop iteration. If there are no more candidate ROIs to evaluate, method 650 proceeds to 660. Trajectory Classification Module Details

[0140] 6F shows a flowchart of an example embodiment of a trajectory classification method 680 that may be performed by trajectory classification module 166 when executed by computer 140. In at least one embodiment, method 680 is performed each time a trajectory transitions from an "active" state to a "finished" state. In an alternative implementation, method 680 is performed over a subset of possible trajectories to reduce processing requirements.

[0141] At this point, all ROIs with sufficiently high pelletness scores are part of the trajectory, but not all of these trajectories are necessarily pellets. Individual objects may look significantly different in different image frames, so an object may look like a pellet in one or more image frames and like a non-pellet object in one or more other image frames. By considering characteristics of the entire trajectory, such as but not limited to the length of the trajectory, the distribution of pelletness scores for the ROIs of the object being tracked within the trajectory, and at least one measure of the statistical distribution of one or more other filter values ​​(e.g., mean, standard deviation, a particular percentile, etc.), the trajectory classification module 166 can make a final classification of each trajectory as being most likely to be associated with a pellet or a non-pellet object.

[0142] At 681, the method includes obtaining data for an ROI within the current trajectory. At 682, trajectory measurements are generated based on the ROI within the current trajectory, including at least one measure of trajectory length, distribution of pelletness scores for the ROI within the trajectory, and / or statistical distribution of one or more other filter values. In at least one embodiment, only the average value of the pelletness scores is used. Alternatively, in at least one embodiment, the average value as well as the 25th / 50th / 75th percentiles are used, which may result in a much clearer picture seen by the pellet classification model.

[0143] At 683, the trajectory measurements are evaluated to determine whether the trajectory from which they were obtained can be classified as a pellet trajectory. In at least one embodiment, this evaluation can be done by providing the trajectory measurements to a trajectory classifier that provides a probability that the trajectory belongs to a pellet trajectory class or a non-pellet trajectory class. This can be done using, for example, logistic regression, a linear model, for example, random forest, or a tree-based model, for example, random forest or XGBoost. Alternatively, in at least one embodiment, this evaluation can be done by comparing at least one trajectory measurement to a corresponding trajectory measurement threshold determined using empirical data. In such an embodiment, if more trajectory measurements pass the trajectory measurement threshold, it is indicated that the current trajectory is likely to be a trajectory of uneaten feed pellets, and the method proceeds to 685, where the trajectory is classified as a pellet trajectory. The method 680 then proceeds to 686.

[0144] If the determination at 683 is false, the method 680 proceeds to 684 where the corresponding object trajectory and the corresponding ROI are discarded.

[0145] At 686, it is determined whether there are any more trajectories to classify. If this determination is true, the method 680 returns to 681. If this determination is not true, the method 680 proceeds to 687 and ends.

[0146] In at least one embodiment, method 600 or a subset thereof further comprises mounting an object enhancer 116 adjacent to the outlet of the discharge pipe 112 facing the feed camera 114 and within the field of view of the feed camera 114 to provide a background for the feed camera 114 so that objects in the captured image have a definable surroundings, and securing the feed camera 114 to the object enhancer 116 such that a video feed captured by the feed camera 114 comprises an image frame that is an image of the object enhancer 116 and includes objects that cross the field of view of the feed camera 114 when the image is captured.

[0147] In at least one embodiment, a system 100 for feed monitoring in a land-based fish farm having an aquarium 110 containing fish comprises a feed camera 114 attached to a discharge pipe 112 coupled to the aquarium 110 and configured to capture a video feed comprising images of objects comprising pellets and non-pellet objects that cross the field of view of the feed camera 114 at the time each image is captured, and a computer 140 having a processor unit 144 and a memory unit 156 having instructions stored thereon that, when executed, cause the processor unit 144 to perform method 600 or a subset of method 600.

[0148] FIG 7A shows a perspective view of an exemplary embodiment of a hardware setup 700 used in the discharge pipe 112. FIG 7B shows a cross-sectional view of the exemplary embodiment shown in FIG 7A. The feed camera 114 may be coupled to a bracket 708 formed from two members 708m1 and 708m2 such that the field of view of the feed camera 114 captures all or most of the object enhancer 116 and there is a fixed distance between the feed camera 114 and the underside of the object enhancer 116. The members 708m1 and 708m2 of the bracket 708 may have an upper end that is a lateral tab that extends outwardly and is coupled to a second bracket 710 having a post 710p to which the underside of the object enhancer 116 is movably coupled. The lower portions of members 708m1 and 708m2 have channels 708c (only one of which is visible) that allow members 708m1 and 708m2 to slide relative to one another, and corresponding lateral tabs are extendable to allow bracket 708 to accommodate outlet pipes 112 having various diameters. Once members 708m1 and 708m2 are correctly positioned so that the lateral extending tabs overlap the edges of outlet pipe 112, fasteners can be applied to channel 708c to secure members 708m1 and 708m2 to one another. Feed camera 114 can be attached to the lower horizontal portions of members 708m1 and 708m2 such that the field of view of feed camera 114 is directed toward object enhancer 116. In at least one implementation of hardware setup 700, the object enhancer 116 is slidably mounted adjacent to the outlet pipe 112 and the feed camera 114 is coupled to the object enhancer 116 at a fixed distance from the object enhancer 116 such that the object enhancer 116 can move up and down as different amounts of water flow out of the outlet pipe 112 and the feed camera 114 is maintained at a fixed distance from the object enhancer 116. In at least one embodiment, the channel 708c may be engaged by a fastener, such as, for example, a screw, to secure the feed camera 114 to the bracket 708.

[0149] The second bracket 710 may be a U-shaped bracket having two vertical members 710v1 and 710v2, each having a tab extending laterally and coupled to a post 710p. The upper lateral tabs of members 710v1 and 710v2 are secured to the upper tabs of bracket 708 via fasteners. Bracket 710 also includes lower members 710m1 and 710m2 coupled to the lower ends of vertical members 710v1 and 710v2. Members 710m1 and 710m2 are movable relative to one another and engaged to one another in a manner similar to that described with respect to the bracket configuration of FIGS. 4A-4B.

[0150] The upper lateral tabs of the second bracket 710 may be secured to the outlet pipe 112 and held in place in a manner that limits the ability of the object enhancer 116 and the feed camera 114 to rotate, but allows the object enhancer 116 to move up and down as water pressure changes in the outlet pipe 112 cause the water to flow out of the outlet pipe 112. The bracket 708, bracket 710, and post 710 may be collectively referred to as a bracket arrangement 740 or mounting assembly, and are shown in isolation in FIG. 7C. FIG. 7D shows the bracket arrangement 740 with the object enhancer 116 in place. The feed camera 114 may have a wire for connecting to a network connection (e.g., an Ethernet cable). Alternatively, the feed camera 114 may have a wireless connection to the Ethernet switch 120.

[0151] For example, the bracket 710 may be attached to the outlet port or opening of the outlet pipe 112 such that the object enhancer 116 is positioned above the outlet port of the outlet pipe 112 and may move upward to provide a gap between the bottom surface of the object enhancer 116 and the top surface of the outlet port to allow water to flow out of the outlet pipe 112 if water pressure increases within the outlet pipe 112. To accomplish this, the bracket 710 includes a sliding mechanism in the form of a post 710p slidably engaged by grooves 116g present in two portions of the edge of the object enhancer 116. If water pressure decreases, the object enhancer 116 may move downward. If water pressure increases, the object enhancer 116 may move upward.

[0152] Although not shown, a retaining element may be used to prevent the object enhancer 116 from dislodging from the post 710p of the bracket 710 and floating out of the exit port of the outlet conduit 112. For example, the end of the post 710p may have a bulbous end or be flared to engage the top of the object enhancer 116 and prevent the object enhancer 116 from dislodging from its mounting (e.g., the post 710p of the bracket 710) at the exit port of the outlet conduit 112 if the water flow is strong enough to lift the object enhancer 116 up to the top of the post 710p.

[0153] Optionally, in at least one embodiment, a light source may be mounted to generate light that is directed toward the object enhancer 116. In such a case, the light source may be secured to the bracket 710.

[0154] One of the technical advantages realized in at least one of the embodiments described herein is that the pellet counting algorithm counts and tracks individual discarded (i.e., leftover) pellets. This is because other approaches either (1) measure the intensity of pellet activity (e.g., detect the number of pellets in a given image frame to represent the relative discard level, but do not actually attempt to measure the number of discarded pellets) or (2) track only a small cross-section of pellets (thus providing a possibly inaccurate estimate). In contrast, the pellet counting algorithm according to the teachings herein accurately counts individual discarded pellets passing through the field of view of the feed camera in real time over a specific time interval, providing an accurate measure rather than an estimate. Depending on the fluid dynamics in the discharge pipe, there may be a small number (possibly negligible number) of pellets that are outside the field of view of the feed camera by adhering to the side of the discharge pipe as they pass through the feed camera. In such a case, the pellet count, after scaling as described above, may not be 100% of the pellets passing through the outlet of the discharge pipe, but may only be slightly lower, for example 95% or 99%. The number of waste pellets (e.g., leftover pellets) passing through the field of view of the feed camera may be a predetermined percentage (e.g., 10% or 20%) of the total number of waste pellets exiting the discharge pipe, which may then be scaled up by a scale factor (e.g., 10 if the predetermined percentage is 10%, 5 if the predetermined percentage is 20%) to extrapolate the total number of waste pellets exiting the aquarium through the discharge pipe, the predetermined percentage being determined by validation as described above.

[0155] Another technical advantage realized in at least one of the embodiments described herein is that the AI-based fish feeding system leverages the infrastructure of a land-based recirculating aquaculture system and provides a significantly more accurate count of waste pellets with minimal computational requirements, since all other approaches applied to conventional systems attempt to create an estimate (rather than an exact count) of the number of waste pellets in the overall aquatic environment. The use of a pellet counting algorithm in addition to utilizing / manipulating the environment (specifically using the outlet pipe of a land-based system) to monitor fish feed within a smaller physical area is unique to the teachings herein and is useful in that all waste feed pellets can be counted with reduced computational requirements (e.g., without the need for one or more GPUs with high computational power typically required for CNNs to process large data sets).

[0156] Another technical advantage realized in at least one of the embodiments described herein is that the pellet counting algorithm allows for a continuous measurement of waste feed, while other methods create discrete measures, because the pellet counting algorithm allows for integration with other real-time data sources to gain a real-time understanding of current feed requirements that can then be used to determine future feed requirements.

[0157] It should be noted that in at least one embodiment, if the computer uses a processor with sufficiently high processing power, such as a GPU, the object enhancer may not be used. In such an embodiment, the ROI proposal module and the ROI classification module may be replaced with various end-to-end object detection neural networks, such as single-shot detectors (SSDs) such as YOLO, and multi-stage models such as Faster-RCNN.

[0158] It should be noted that in at least one embodiment, the pellet counter algorithm may be further configured to generate a control signal based on the pellet count and send the control signal to a fish feeder at the fish farm to generate a new amount of fish feed to be fed to the aquarium. The control signal may be updated at intervals proportional to the interval used to count the pellets, or the control signal may be updated instantly when there is a change in the gradient of uneaten fish feed (i.e. uneaten feed pellets) that exceeds a threshold set to indicate that there may be a recent problem in the aquarium at the fish farm.

[0159] While applicants' teachings described herein relate to various embodiments for illustrative purposes, it is not intended that applicants' teachings be limited to such embodiments, as the embodiments described herein are intended to be exemplary. On the contrary, applicants' teachings described and illustrated herein encompass various alternatives, modifications, and equivalents without departing from the embodiments described herein, the general scope of which is defined in the appended claims.

Claims

1. 1. A system for feed monitoring in a land-based fish farm, comprising an aquarium containing fish and a discharge pipe coupled to the aquarium, wherein uneaten feed pellets and non-pellet objects flow through the discharge pipe to an exit port, the system comprising: a feed camera configured to capture a video feed comprising a plurality of images of the interior of the outlet pipe including objects that cross the field of view of the feed camera as each image is captured, the objects comprising the uneaten feed pellets and the non-pellet objects; and 1. A computing device comprising at least one processor and a memory unit in the form of a non-transitory computer readable medium having instructions stored thereon, the instructions, when executed, causing the at least one processor to: acquiring a number of images from said video feed at a given time interval; pre-processing the image to obtain candidate regions of interest (ROIs), each ROI corresponding to a unique object in the image, the object being likely to be a feed pellet; performing classification on the candidate ROIs and obtaining a pelletness score representing the probability that the object corresponding to the candidate ROI belongs to a feed pellet class; determining whether to retain a candidate ROI based on a higher probability that the retained candidate ROI belongs to the feed pellet class; assigning each of the retained candidate ROIs to a trajectory; classifying the trajectory as a pellet trajectory or a non-pellet trajectory; generating a pellet count in real time based on the pellet trajectory, the pellet count representing a predetermined percentage of uneaten feed pellets in the aquarium that have not been consumed by the fish; a computing device for causing the A system comprising:

2. 2. The system of claim 1, wherein the at least one processor is further configured to transmit the pellet count to the fish farm, and the pellet count is used by a fish feeder to generate a new amount of fish food to supply to the aquarium based at least in part on the pellet count.

3. 2. The system of claim 1, wherein the at least one processor is further configured to generate a control signal based on the pellet count and send the control signal to a fish feeder in the fish farm to generate a new amount of fish feed to be fed to the aquarium.

4. The system of claim 1 , wherein the at least one processor is further configured to determine a slope of the uneaten feed pellets over the time interval and generate an alert if the slope is greater than a slope threshold.

5. 2. The system of claim 1, wherein the at least one processor is configured to obtain candidate ROIs in the given image frame by performing background subtraction on the given image frame to identify any objects likely to change position in subsequent image frames, performing contour detection on each identified object to define a boundary that is a potential ROI for the identified object, and filtering the potential ROIs to obtain the candidate ROIs.

6. 2. The system of claim 1, wherein the at least one processor is configured to classify the candidate ROI by generating, for each candidate ROI, samples where a given sample includes image data of the corresponding candidate ROI, pre-processing the samples, and applying an ROI classifier to each pre-processed sample to generate a probability distribution having a probability that the sample belongs to different classes.

7. The system of claim 6 , wherein the classes include a pellet class, a non-pellet class, and an ambiguous class.

8. The system of claim 7 , wherein the pelletness score represents a probability that the sample belongs to the pellet class.

9. 7. The system of claim 6, wherein the at least one processor is configured to preprocess the samples by resizing the samples to have the same matrix size and performing mean-center normalization.

10. 2. The system of claim 1, wherein the at least one processor is configured to assign each of the candidate ROIs to one of the trajectories within one image frame by: (a) assigning the candidate ROI to another new trajectory if there is no active trajectory; (b) assigning the candidate ROI to the nearest active trajectory if a distance between a position of the candidate ROI and a position of a last ROI in the nearest active trajectory is less than a distance threshold; or (c) assigning the candidate ROI to a new active trajectory if a distance between a position of the candidate ROI and a position of a last ROI in the nearest active trajectory is greater than a distance threshold if there is an active trajectory.

11. 2. The system of claim 1, wherein the at least one processor is configured to classify the trajectories into pellet trajectories or non-pellet trajectories by identifying terminated trajectories including trajectories that have not been updated for one of the images in the given time period, determining at least one trajectory measurement for each of the terminated trajectories, and determining which terminated trajectories belong to the pellet trajectory class based on the at least one trajectory measurement.

12. 12. The system of claim 11, wherein the at least one processor is configured to determine which terminated trajectories belong to the pellet trajectory class by comparing the at least one trajectory measurement to a threshold or providing the at least one trajectory measurement to a trajectory classifier.

13. 12. The system of claim 11, wherein the at least one trajectory measurement comprises at least one measure of a length of the terminated trajectory, a distribution of pelletness scores of ROIs that comprise the terminated trajectory, and / or a statistical distribution of one or more other filter values.

14. 1. A computer-implemented method for feed monitoring in tanks of a land-based fish farm, the method comprising: reading image frames and associated metadata from a video feed captured from a feed camera at a given time interval, the image frames captured by the feed camera being images of uneaten feed pellets and non-pellet objects being expelled from the aquarium; pre-processing the image frames to obtain candidate regions of interest (ROIs), each corresponding to a unique object in the image, the object being likely to be a feed pellet; performing classification on the candidate ROIs to obtain a pelletness score corresponding to the candidate ROI that represents the probability that the object belongs to a feed pellet class; determining, using program instructions stored in a memory unit, whether to retain the candidate ROI based on a probability that the candidate ROI belongs to the feed pellet class; assigning each of the retained candidate ROIs to a trajectory; classifying the trajectory as a pellet trajectory or a non-pellet trajectory; generating a pellet count in real time based on the pellet trajectory, the pellet count representing a predetermined percentage of uneaten feed pellets in the aquarium that have not been consumed by the fish; A method comprising:

15. 15. The method of claim 14, further comprising transmitting the pellet count to the fish farm, wherein the pellet count is used by a fish feeder to generate a new amount of fish food to feed to the aquarium based at least in part on the pellet count.

16. 15. The method of claim 14, further comprising: generating a control signal based on the pellet count; and sending the control signal to a fish feeder in the fish farm to generate a new amount of fish food to feed to the aquarium.

17. 15. The method of claim 14, further comprising: determining gradients of the uneaten feed pellets in the given time interval; and generating an alert if one of the gradients is greater than a gradient threshold.

18. 15. The method of claim 14, wherein obtaining the candidate ROIs in a given image frame comprises: performing background subtraction on the given image frame to identify any objects that change position in subsequent image frames; performing contour detection on each identified object to define a boundary that is a potential ROI for the identified object; and filtering the potential ROIs to obtain the candidate ROIs.

19. 15. The method of claim 14, wherein performing classification on the candidate ROIs comprises: generating, for each candidate ROI, samples, where a given sample includes image data of the corresponding candidate ROI; pre-processing the samples; and applying an ROI classifier to each pre-processed sample to generate a probability distribution having probabilities that the sample belongs to different classes.

20. The method of claim 19 , wherein the classes include a pellet class, a non-pellet class, and an ambiguous class.

21. 21. The method of claim 20, wherein the pelletness score represents a probability that the sample belongs to the pellet class.

22. The method of claim 14 , wherein the method comprises pre-processing the samples by resizing the samples to have the same matrix size and performing mean-center normalization.

23. 15. The method of claim 14, wherein assigning each of the candidate ROIs to one of the trajectories within one image frame comprises: (a) assigning the candidate ROI to another new trajectory if there is no active trajectory; (b) assigning the candidate ROI to a nearest active trajectory if a distance between a position of the candidate ROI and a position of a last ROI in the nearest active trajectory is less than a distance threshold; or (c) assigning the candidate ROI to a new active trajectory if a distance between a position of the candidate ROI and a position of a last ROI in the nearest active trajectory is greater than a distance threshold if there is an active trajectory.

24. 15. The method of claim 14, wherein classifying the trajectories into pellet trajectories or non-pellet trajectories comprises identifying terminated trajectories including trajectories that have not been updated for one of the images in the given time period; determining at least one trajectory measurement for each of the terminated trajectories; and determining which terminated trajectories belong to the pellet trajectory class based on the at least one trajectory measurement.

25. 25. The method of claim 24, wherein determining which terminated trajectories belong to the pellet trajectory class comprises comparing the at least one trajectory measurement to a threshold or providing the at least one trajectory measurement to a trajectory classifier.

26. 25. The method of claim 24, wherein the at least one trajectory measurement comprises at least one measure of a length of the terminated trajectory, a distribution of pelletness scores of ROIs that comprise the terminated trajectory, and / or a statistical distribution of one or more filter values.

27. 1. A system for feed monitoring in a land-based fish farm having tanks containing fish, the system comprising: a feed camera attached to an outlet pipe coupled to the aquarium, the feed camera configured to capture a video feed comprising a plurality of images of objects that cross a field of view of the feed camera as each image is captured, the objects comprising pellets and non-pellet objects; and A computing device comprising at least one processor and a non-transitory computer readable medium having instructions stored thereon, the instructions, when executed, causing the at least one processor to perform the method of claim 14; A system comprising:

Images