Systems and methods for fish volume estimation, weight estimation, and analytic value generation

Abstract

To provide accurate and timely monitoring of fish in a fish pen deployed in open water, in order to efficiently manage operation of the fish pen.SOLUTION: In some embodiments, an underwater camera system for monitoring fish in a fish pen is provided. In some embodiments, a stereoscopic camera may be used to determine dimensions of fish, and the underwater camera system may use the fish dimensions to estimate biomass of fish in the fish pen. The biomass value and rates of change of the biomass value may be used to adjust feeding of the fish in the fish pen. In some embodiments, images captured by the stereoscopic camera may be used to focus a variable focal lens camera on a fish to obtain high-resolution images that can be used for diagnosing fish conditions. In some embodiments, images captured by a camera may be used to predict motion of the fish, and the predicted motion may be used to generate various fish monitoring analytic values.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0002] In some embodiments, a computer-implemented method for determining the biomass of at least one fish in a fish pen is provided. A computing device detects the location of the fish in an image captured by an underwater stereoscopic camera. The computing device determines a segmentation mask of the fish in the image. The computing device determines whether the segmentation mask is associated with a measurable fish. In response to determining that the segmentation mask is associated with the measurable fish, the computing device determines at least one dimension of the fish based on the image, determines a weight of the fish based on the at least one dimension of the fish, and adds the weight of the fish to a set of fish weight measurements.

[0003] In some embodiments, an underwater camera system for monitoring a plurality of fish in a fish pen is provided. The system includes at least one camera and a computing device communicatively coupled to the at least one camera. The computing device is configured to detect fish in at least one image from the at least one camera, determine movement behavior of the fish from the at least one image, and determine at least one fish-monitoring analytic value based at least in part on the movement behavior of the fish.

[0004] In some embodiments, a non-transitory computer-readable medium having stored thereon computer-executable instructions that, upon execution by one or more processors of the computing device, cause the computing device to perform actions for monitoring a plurality of fish in a fish pen, the actions including receiving, by the computing device, an image captured by an underwater camera system showing at least one fish, determining, by the computing device, measurability of the at least one fish in the image based at least in part on whether the at least one fish is occluded, and, in response to determining that the at least one fish in the image is measurable, determining at least one characteristic of the at least one fish based on the image.

[0005] In some embodiments, an underwater camera system for monitoring at least one fish in a fish pen is provided. The system includes a stereoscopic camera, a variable focus lens camera, and a computing device. The computing device is communicatively coupled to the variable focus lens camera and the stereoscopic camera. The computing device is configured to detect a fish in images from the stereoscopic camera, adjust a focus of the variable focus lens camera based on the images from the stereoscopic camera, capture a focused image of the fish using the variable focus lens camera, and determine at least one fish-monitoring analytic value based at least in part on the focused image. [Brief explanation of the drawings]

[0006] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which that element is first introduced.

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating a non-limiting exemplary embodiment of a system for monitoring and controlling conditions within a fish pen, according to various aspects of the present disclosure. [Figure 2]FIG. 1 is a block diagram illustrating a non-limiting exemplary embodiment of an underwater camera system according to various aspects of the present disclosure. [Figure 3] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining the total biomass of a plurality of fish in a fish pen, according to various aspects of the present disclosure. [Figure 4] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a procedure for determining the measurability of a fish associated with a segmentation mask in an image, according to various aspects of the present disclosure. [Figure 5] 1A-1C are images illustrating non-limiting exemplary embodiments of occluded and unoccluded segmentation masks and images, according to various aspects of the present disclosure; [Figure 6] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a procedure for measuring the length of a fish based on an image and a segmentation mask, according to various aspects of the present disclosure. [Figure 7] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining at least one fish-monitoring analytic value according to various aspects of the present disclosure. [Figure 8] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining at least one fish-monitoring analytic value according to various aspects of the present disclosure. [Figure 9] FIG. 1 is a block diagram illustrating a non-limiting, exemplary embodiment of a computing device suitable for use as a computing device with embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] Aquaculture techniques, such as fish farming, are becoming increasingly popular as a method of raising fish for consumption. One relatively new form of aquaculture uses fish pens deployed in open water. While such systems provide a beneficial environment for raising fish, the underwater deployment of the fish pens makes it difficult to monitor conditions within the pens and the health of the fish. To efficiently manage fish pen operations, accurate and timely monitoring of the fish within the pens is desirable. Weight and health are two conditions for which accurate monitoring is desirable: monitoring changes in biomass within a fish pen can inform decisions regarding whether to increase or decrease the feeding system it supports, and monitoring the health of the fish within a fish pen can inform decisions regarding whether to provide treatment for any conditions afflicting the fish.

[0009] FIG. 1 is a schematic diagram illustrating a non-limiting exemplary embodiment of a system for monitoring and controlling conditions within a fish pen, according to various aspects of the present disclosure.

[0010] In the illustrated system 100, the aquaculture management computing device 102 is communicatively coupled to an underwater camera system 106 installed with a fish pen 104. In some embodiments, the underwater camera system 106 is configured to capture and analyze images of the fish in the fish pen 104 to determine at least one of fish health information and biomass estimates.

[0011] In some embodiments, the underwater camera system 106 transmits at least one of the health information and the biomass estimate to the aquaculture management computing device 102. In some embodiments, the aquaculture management computing device 102 may present the received information to a user so that the user can take action to manage the fish pen 104, including, but not limited to, changing feeding rates and providing treatment for poor health. In some embodiments, the aquaculture management computing device 102 may receive the information and autonomously take management actions with or without presenting the information to a user, including, but not limited to, automatically changing feeding rates with an automatic feed pellet dispenser, automatically changing the depth of the fish pen 104, and automatically dispensing medication.

[0012] Although the aquaculture management computing device 102 is shown in FIG. 1 as a laptop computing device, in some embodiments, other types of computing devices may be used for the aquaculture management computing device 102, including, but not limited to, a desktop computing device, a server computing device, a tablet computing device, a smartphone computing device, and one or more computing devices in a cloud computing system.

[0013] 1 illustrates an exemplary embodiment in which the underwater camera system 106 is installed with a fish pen 104, which is an underwater fish pen that may be submerged in an open body of water, such as the open ocean. However, this embodiment should not be considered limiting, and in some embodiments, the underwater camera system 106 may be installed underwater in different environments. For example, in some embodiments, the underwater camera system 106 may be installed in a floating fish pen at an aquaculture site near a shoreline. As another example, in some embodiments, the underwater camera system 106 may be utilized in an underwater fish pen associated with a land-based hatchery or farm to monitor fish larvae, hatchlings, and / or fingerlings. As yet another example, in some embodiments, the underwater camera system 106 may be utilized in a fish pen associated with a recirculating aquaculture system (RAS), in which land-based fish pens coupled to advanced water treatment systems are used to grow fish.

[0014] 2 is a block diagram illustrating a non-limiting exemplary embodiment of an underwater camera system 106 according to various aspects of the present disclosure. The underwater camera system 106 is designed to provide a high precision system for monitoring fish within a fish pen to determine characteristics such as health and total biomass.

[0015] As shown, underwater camera system 106 includes a stereo camera 202, a variable focus lens camera 204, and a computing device 206. In some embodiments, a housing (not shown) may enclose components of underwater camera system 106, or may enclose some components of underwater camera system 106 and provide external mounting locations for other components of underwater camera system 106.

[0016] In some embodiments, the stereoscopic camera 202 is a camera that includes two cameras that are physically offset from one another so that parallax can be used to determine depth information for pixels in an image. In some embodiments, cameras using other depth-sensing technologies, including but not limited to time-of-flight sensors, may be used in place of the stereoscopic camera 202. In some embodiments, the variable focus lens camera 204 is a camera that can capture high-resolution photographs over a limited depth of focus. Because the depth of focus is limited, the variable focus lens camera 204 has a mechanism for adjusting the distance of the focus from the variable focus lens camera 204. In general, any suitable stereoscopic camera 202 and variable focus lens camera 204 adapted for underwater use may be used. In some embodiments, only one or the other of the stereoscopic camera 202 and variable focus lens camera 204 may be present. In some embodiments, the underwater camera system 106 may include a wide-field camera instead of or in addition to the stereoscopic camera 202.

[0017] As shown, computing device 206 includes one or more processors 208, a network interface 210, and a computer-readable medium 212. In some embodiments, processor 208 may include one or more commercially available general-purpose computer processors, each of which may include one or more processing cores. In some embodiments, processor 208 may also include one or more special-purpose computer processors, including one or more processors adapted to efficiently perform machine learning tasks and / or one or more processors adapted to efficiently perform computer vision tasks, including, but not limited to, a Compute Unified Device Architecture (CUDA)-enabled graphics processing unit (GPU).

[0018] In some embodiments, the network interface 210 may implement any suitable communication technology for communicating information from the underwater camera system 106 to another computing device. For example, in some embodiments, the network interface 210 may include a wireless interface implementing 2G, 3G, 4G, 5G, LTE, Wi-Fi, WiMAX, Bluetooth, satellite-based, or other wireless communication technology. As another example, in some embodiments, the network interface 210 may include a wired interface implementing Ethernet, USB, Firewire, CAN-BUS, or other wired communication technology. In some embodiments, the network interface 210 may include multiple communication technologies. For example, the network interface 210 may include a wired interface coupling the underwater computing device 206 to a transceiver located on a buoy or other surface component, and the transceiver may use wireless communication technology to communicate with an on-board or land-based computing device. In some embodiments, the underwater camera system 106 may transfer data via the exchange of removable computer-readable media, including, but not limited to, flash memory or a hard drive.

[0019] In some embodiments, computer-readable medium 212 is installed on computing device 206 and stores computer-executable instructions that, in response to execution by processor 208, cause underwater camera system 106 to provide various engines and / or data stores. In some embodiments, computer-readable medium 212 may include magnetic computer-readable medium, including but not limited to a hard disk drive. In some embodiments, computer-readable medium 212 may include optical computer-readable medium, including but not limited to a DVD or CD-ROM drive. In some embodiments, computer-readable medium 212 may include electronic non-volatile storage medium, including but not limited to flash memory. In some embodiments, at least some of the computer-executable instructions may be programmed into a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other device that combines storage of computer-executable instructions with one or more circuits for implementing logic. In some embodiments, such an FPGA or ASIC may provide aspects of both computer-readable medium 212 and computing device 206. In some embodiments, a combination of the above computer-readable media may be provided to collectively provide the illustrated computer-readable medium 212 .

[0020] As shown, computer-readable medium 212 includes image analysis engine 220, fish measurement engine 222, fish measurement data store 214, fish weight data store 216, and machine learning model data store 218. In some embodiments, image analysis engine 220 receives and processes information from stereo camera 202 and / or variable focus lens camera 204 to capture images of fish in fish pen 104 for analysis. In some embodiments, image analysis engine 220 also performs pre-processing, filtering, and / or other analysis of the images. In some embodiments, image analysis engine 220 may send control commands to stereo camera 202 and / or variable focus lens camera 204. In some embodiments, fish measurement engine 222 receives images processed by image analysis engine 220 and generates measurements of fish in the images. Further description of actions performed by image analysis engine 220 and fish measurement engine 222 is provided below.

[0021] In some embodiments, fish measurement data store 214 stores fish measurements generated by fish measurement engine 222 and / or image analysis engine 220. In some embodiments, fish weight data store 216 stores correlations between fish characteristics determinable by image analysis engine 220, such as one or more dimensions of the fish, and fish weights. In some embodiments, machine learning model data store 218 stores one or more machine learning models trained to perform various tasks on images. For example, machine learning model data store 218 may store one or more convolutional neural networks trained to detect fish in images. Further description of the information stored by and retrieved from fish measurement data store 214, fish weight data store 216, and machine learning model data store 218 is provided below.

[0022] As used herein, an "engine" refers to logic embodied in hardware or software instructions, which may be written in a programming language such as C, C++, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft .NET™, Go, or Python. An engine may be compiled into an executable program or written in an interpreted programming language. A software engine may be called from other engines or from itself. Generally, an engine as described herein refers to a logical module that can be merged with other engines or divided into sub-engines. An engine may be implemented by logic stored on any type of computer-readable medium or computer storage device and stored and executed on one or more general-purpose computers, thus creating an engine or a special-purpose computer configured to provide its functionality. An engine may be implemented by logic programmed into an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another hardware device.

[0023] Fish measurement data store 214, fish weight data store 216, and machine learning model data store 218 are shown as residing on computer readable medium 212. In some embodiments, one or more of fish measurement data store 214, fish weight data store 216, and machine learning model data store 218 may reside on different computer readable media or different computing devices.

[0024] As used herein, a "data store" refers to any suitable device configured to store data for access by a computing device. One example of a data store is a reliable, high-speed relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technology and / or device capable of quickly and reliably providing stored data in response to a query may be used, and the computing devices may be accessible locally rather than over a network, or may be provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, such as a hard disk drive, flash memory, RAM, ROM, or any other type of computer-readable storage medium. Those skilled in the art will recognize that the separate data stores described herein can be combined into a single data store and / or the single data store described herein can be separated into multiple data stores without departing from the scope of the present disclosure.

[0025] FIG. 3 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining the biomass of at least one fish in a fish pen, according to various aspects of the present disclosure. Measuring fish in a fish pen using camera images in an automated manner faces several challenges. For example, poor image quality due to focus / motion blur, occlusion of fish by other fish or fish pen hardware, poor lighting, or fish angles can produce poor monitoring data. In some embodiments, method 300 attempts to avoid these problems by providing a quality determination pipeline and unique measurement techniques to gather data from the camera images that is accurate enough to generate a reliable estimate of the biomass in the fish pen. In some embodiments, measurements of multiple fish may be collected and used, along with an estimate of the fish population in the fish pen, to determine the total biomass of the fish in the fish pen.

[0026] From a start block, the method 300 proceeds to block 302, where the stereo camera 202 of the underwater camera system 106 captures an image of a fish. In some embodiments, the image captured by the stereo camera 202 may include more than one fish, and the method 300 can process multiple fish in a single image. However, for ease of explanation, the description of the method 300 will assume that the image includes a single fish to be measured. The image captured by the stereo camera 202 includes color and depth information for each pixel in the image.

[0027] In block 304, the image analysis engine 220 of the underwater camera system 106 determines the location of the fish in the image. In some embodiments, the image analysis engine 220 may use a first machine learning model from the machine learning model data store 218 to determine a bounding box that defines the location of the fish in the image. In some embodiments, the first machine learning model used to determine the location of the fish in the image may be a convolutional neural network trained on other images of fish to detect fish in stereo images.

[0028] At block 306, the image analysis engine 220 determines a segmentation mask of the fish in the image. In some embodiments, the image analysis engine 220 may use a second machine learning model from the machine learning model data store 218 to determine the segmentation mask. For example, the image analysis engine 220 may crop out a portion of the image that includes the location of the fish determined at block 304 and provide the cropped portion as input to the second machine learning model. In some embodiments, the second machine learning model may be another convolutional neural network. In some embodiments, other techniques, including, but not limited to, edge detection and depth discontinuity detection, may be used to determine or refine the segmentation mask of the fish.

[0029] At process block 308, the image analysis engine 220 determines whether the segmentation mask is associated with a measurable fish. One problem with automated processing of images from an underwater camera is that many of the images captured by the underwater camera may be unsuitable for automated analysis. Thus, the method 300 may use any suitable procedure at process block 308 to determine whether the image is suitable for measuring the fish associated with the segmentation mask. One non-limiting example of such a procedure is shown in FIG. 4 and described in detail below.

[0030] The method 300 then proceeds to decision block 310 where a decision is made based on whether the fish was determined to be measurable. If the fish was not determined to be measurable, the result of decision block 310 is NO and the method 300 returns to block 302 to capture another image of the fish. Otherwise, if the fish was determined to be measurable, the result of decision block 310 is YES and the method 300 proceeds to procedure block 312.

[0031] At process block 312, the fish measurement engine 222 of the underwater camera system 106 determines at least one dimension of the fish based on the image and the segmentation mask, and at block 314, the fish measurement engine 222 determines the weight of the fish by querying the fish weight data store 216 of the underwater camera system 106 using the at least one dimension of the fish. In some embodiments, all of the fish in the fish pen 104 may be the same, and thus the fish weight data store 216 may be queried without having to individually determine the species of the fish. In some embodiments, multiple different species of fish may be present in the fish pen 104, and thus an additional step of determining the species of the fish may be performed (such as providing the segmentation mask to another machine learning model stored in the machine learning model data store 218) to obtain the species of the fish indicated for query.

[0032] Method 300 may determine any dimension (or dimensions) that can be used to query fish weight data store 216 for fish weight. For example, data correlating the length and / or height of a given species of fish with body mass may be available (and stored in fish weight data store 216). Thus, the weight of the fish may be determined by querying fish weight data store 216 using the fish's length and / or height determined in process block 312. Any suitable technique for obtaining at least one dimension of the fish based on the image and segmentation mask may be used. One non-limiting example of such a technique is shown in FIG. 6 and described in further detail below.

[0033] At block 316, the fish measurement engine 222 adds the weight of the fish to the fish measurement data store 214 of the underwater camera system 106. In some embodiments, the fish measurement engine 222 may also store the measured dimensions of the fish in the fish measurement data store 214.

[0034] The method 300 then proceeds to decision block 318 where a determination is made as to whether enough measurements of fish have been added to the fish measurement data store 214. In some embodiments, the total number of fish measurements added to the fish measurement data store 214 may be compared to a threshold number of measurements. In some embodiments, a confidence interval may be configured and used to determine the threshold number of measurements. For example, if the fish pen 104 is known to contain 40,000 fish, configuring the underwater camera system 106 to use a 95% confidence interval means that approximately 381 fish measurements would need to be collected to ensure that the true average weight of all fish in the pen is within 5% of the sample average.

[0035] If more measurements are needed to construct a statistically relevant sample of the fish in the fish pen 104, the result of decision block 318 is NO and the method 300 returns to block 302 to capture another image of the fish. Otherwise, if enough measurements have been stored in the fish measurement data store 214, the result of decision block 318 is YES and the method 300 proceeds to block 320.

[0036] In block 320, the fish measurement engine 222 determines the fish biomass using the weights stored in the fish measurement data store 214. In some embodiments, the fish measurement engine 222 may determine an average of the stored fish measurements and multiply that average by the number of fish in the fish pen 104 to determine the total fish biomass. In some embodiments, the fish measurement engine 222 may determine the number of fish in the fish pen 104 based on the initial number of fish counted as added to the fish pen 104 minus a mortality count, which reflects the number of fish known to have died, and an estimated loss count, which reflects the number of fish removed from the fish pen 104 by undetected means (including, but not limited to, external predation). In some embodiments, the fish measurement engine 222 may use the average of the stored fish measurements themselves as the determined fish biomass.

[0037] The method 300 then proceeds to an end block and ends. Although FIG. 3 shows the method 300 stopping at this point, in some embodiments, the method 300 can continue to use the determined total fish biomass for any purpose. For example, in some embodiments, the fish measurement engine 222 can transmit the total fish biomass to the aquaculture management computing device 102, which can then present the total fish biomass to a user. In some embodiments, the fish measurement engine 222 can transmit the total fish biomass to the aquaculture management computing device 102, which can then use the total fish biomass to control an automatic feeding device. As another example, the fish measurement engine 222 can transmit an average of the stored fish measurements to be used for similar purposes.

[0038] In some embodiments, the aquaculture management computing device 102 may measure the rate of change of fish biomass over time, which may be used to control the fish feeding device. In some embodiments, the biomass estimate and growth rate trend may be used to select feed pellet size and / or feeding type for the population in the fish pen 104. The biomass estimate and growth rate trend may also be used to set the total feed amount, feed delivery style, feeding rate, and / or feeding time. In some embodiments, the biomass estimate and / or the rate of change of total fish biomass over time may be input to a feedback loop mechanism for controlling the automatic feeding device. Other inputs to the feedback loop mechanism may include one or more of a set of environmental data, such as water temperature and water pressure (distance below the surface). The output of the feedback loop mechanism may indicate entry / exit for feed delivery on a given day.

[0039] FIG. 4 is a flowchart illustrating a non-limiting, exemplary embodiment of a procedure for determining the measurability of a fish associated with a segmentation mask in an image, according to various aspects of the present disclosure. Procedure 400 is a non-limiting example of a procedure suitable for use in procedure block 308 of FIG. 3. Generally, procedure 400 provides an image quality estimation pipeline that allows images that are likely to result in inaccurate fish weight estimates to be ignored. In some embodiments, procedure 400 operates on a cropped portion of the image that has been determined by image analysis engine 220 to contain a fish, as described above. In some embodiments, procedure 400 may also receive segmentation mask information that indicates pixels of the image associated with the fish to be analyzed. As described above, because the images are captured by stereoscopic camera 202, the images include both color and depth information for each pixel.

[0040] From a start block, procedure 400 proceeds to block 402, where the image analysis engine 220 determines the amount of blur associated with the image of the fish. Blur may be detected using any suitable technique, including, but not limited to, determining the variation of the Laplacian, computing a fast Fourier transform to examine the distribution of high and low frequencies, using a machine learning model trained on blurred and non-blurred images, or any other suitable technique.

[0041] The procedure 400 then proceeds to decision block 404, where a decision is made based on the amount of blur determined by the image analysis engine 220. In some embodiments, the decision may be made by comparing the amount of blur to a predetermined threshold amount of blur that depends on the blur detection technique used.

[0042] If the comparison of the amount of blur to the predetermined threshold amount of blur indicates that the image of the fish is too blurred, the result of decision block 404 is YES and procedure 400 proceeds to block 418. Otherwise, if the comparison of the amount of blur to the predetermined threshold amount of blur does not indicate that the image of the fish is too blurred, the result of decision block 404 is NO and procedure 400 proceeds to block 406.

[0043] In block 406, the image analysis engine 220 processes the segmentation mask using a shape analysis filter. In some embodiments, the shape analysis filter is trained to extract features from the shape of the segmentation mask and compare those features to features extracted from a training set of known good segmentation masks. In some embodiments, a set of Zernike moments may be determined for the segmentation mask. Zernike moments contain features that are orthogonal to each other, thereby minimizing information redundancy. Zernike moment-based features are also translation- and scale-invariant, thus providing a rigorous shape-matching criterion. In some embodiments, the segmentation mask may be standardized to remove scale or translation invariance by rotating, tight-cropping, and / or resizing the segmentation mask to fixed dimensions before extracting features.

[0044] The procedure 400 then proceeds to decision block 408, where a decision is made based on whether the shape analysis filter determined that the features extracted from the segmentation mask are sufficiently similar to the features extracted from the known good segmentation mask. If the shape analysis filter determines that the segmentation masks are not sufficiently similar, the result of decision block 408 is YES and the procedure 400 proceeds to block 418. Otherwise, if the shape analysis filter determines that the segmentation mask has a shape similar to the known good segmentation mask shape, the result of decision block 408 is NO and the procedure 400 proceeds to block 410.

[0045] One common problem when using cameras to monitor fish in a fish pen 104 is that as the fish grow and begin to crowd the fish pen 104, it becomes increasingly common for one fish's view to be occluded by another. FIG. 5 is an image illustrating non-limiting, exemplary embodiments of occlusion segmentation masks and images, as well as non-limiting, exemplary embodiments of unoccluded segmentation masks and images, according to various aspects of the present disclosure. In a first image 502, an unoccluded fish 508 is visible to the camera for its entire length. Thus, a corresponding unoccluded segmentation mask 506 shows the entire unoccluded fish 508. In a second image 504, a portion of an occluded fish 512 is visible to the camera, but an occluding fish 514 closer to the camera has a fin that covers the front of the occluded fish 512. Thus, a corresponding occlusion segmentation mask 510 is truncated at the front of the occluded fish 512. However, the shape of the occlusion segmentation mask 510 is still fish-like, and therefore, despite the occlusion, the shape analysis filter applied in block 406 may not have determined that the second image 504 is not measurable. Additional analysis needs to be performed to detect occlusions in the image that make the segmentation mask unsuitable for measurement.

[0046] To that end, returning to block 410 of FIG. 4 , image analysis engine 220 determines the average depth of pixels in a region within the segmentation mask, and in block 412, image analysis engine 220 compares the depth of pixels in a region outside the segmentation mask to the average depth of pixels in a region within the segmentation mask. As shown in FIG. 5 , image analysis engine 220 may determine the average depth of pixels in a region a predetermined number of pixels outside the boundary of the segmentation mask and determine the average depth of pixels in a region a predetermined number of pixels inside the boundary of the segmentation mask (or the entire segmentation mask). In some embodiments, image analysis engine 220 may first partition the segmentation mask into slices (e.g., vertical, horizontal, or radial slices) and separately determine the average depth of pixels inside and outside the boundary of the segmentation mask within each slice. Image analysis engine 220 then compares the average depth of pixels in the outer portion with the average depth of pixels in the inner portion. If the fish is not occluded, the average depth of the pixels in the outer portion (or the outer portion per slice) is greater than the average depth of the pixels in the inner portion. However, if the fish is occluded, the average depth of the pixels in the outer portion (or the outer portion of at least one slice) is greater than the average depth of the pixels in the inner portion, indicating the presence of an object closer to the camera than the fish. In Figure 5, all pixels surrounding the segmentation mask of the unoccluded fish 508 in the first image 502 have a greater depth than the pixels in the segmentation mask, but the pixels surrounding the segmentation mask of the occluded fish 512 in the slice in front of the occluded fish 512 will have a lesser depth than the pixels in the segmentation mask, thus indicating the presence of an occluding fish 514.

[0047] Procedure 400 then proceeds to decision block 414, where it determines whether the object identified by the segmentation mask is occluded using the comparison performed in block 412. If the comparison indicates that the object is occluded, the result of decision block 414 is YES, and procedure 400 proceeds to block 418. Block 418 is the target of each decision block result indicating that the image of the fish is not suitable for measurement. Thus, in block 418, image analysis engine 220 determines that the fish is not measurable. Procedure 400 then proceeds to an end block and ends.

[0048] Returning to decision block 414, if the comparison indicates that the object is not occluded, the result of decision block 414 is NO and procedure 400 proceeds to block 416. At block 416, all tests on the image of the fish have been performed and none of the tests indicate that the fish in the image is not measurable. Therefore, at block 416, image analysis engine 220 determines that the fish is measurable. Procedure 400 then proceeds to an end block and ends.

[0049] Upon completion, procedure 400 provides its caller with a determination of whether the fish is measurable.

[0050] 6 is a flowchart illustrating a non-limiting example embodiment of a procedure for measuring the length of a fish based on an image and a segmentation mask, according to various aspects of the present disclosure. Procedure 600 is an example of a technique suitable for use in process block 312 of FIG. 3, where length is at least one dimension to be determined.

[0051] From a start block, procedure 600 proceeds to block 602, where image analysis engine 220 separates pixels into slices along the axes of the segmentation mask. For example, image analysis engine 220 may separate the segmentation mask into slices that are a given number of pixels wide. The number of pixels may be configurable to manage the tradeoff between computational speed and fine-grained accuracy. As a non-limiting example, in some embodiments, image analysis engine 220 may separate the segmentation mask into slices that are 10 pixels wide, although any other suitable number of pixels may be used.

[0052] In block 604, the image analysis engine 220 determines a depth value for each slice. In some embodiments, the image analysis engine 220 may determine the average depth of the pixels in each slice and use the average depth as the depth of the slice. In some embodiments, the image analysis engine 220 may determine the depth of the pixel at the midpoint of each slice and use the depth of the midpoint pixel as the depth of the slice. In some embodiments, the image analysis engine 220 may determine the depth of the pixel at each boundary of each slice.

[0053] At block 606, the image analysis engine 220 generates a curvilinear representation of the fish based on the depth values of the slices, i.e., in some embodiments, the image analysis engine 220 can determine the position of each of the slices in three-dimensional space relative to the camera.

[0054] In block 608, the image analysis engine 220 determines the length of the curvilinear representation using a pinhole model. That is, the image and depth values may be processed as if the camera that captured the image were a pinhole camera, and the image analysis engine 220 may use the distance of each slice from the camera, along with the horizontal width of each slice, to calculate the actual length in three-dimensional space of the curvilinear representation of the fish. In some embodiments, the image analysis engine 220 may also compensate for underwater refraction as part of its pinhole model.

[0055] Procedure 600 then proceeds to an end block and terminates. Upon termination, procedure 400 provides its determination of the fish's length to its caller. It will be appreciated that a similar technique can be used to determine the fish's height by simply reorienting the process vertically rather than horizontally.

[0056] 7 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining at least one fish monitoring analytic value according to various aspects of the present disclosure. In addition to measuring fish dimensions and using the dimensions to determine total fish weight and biomass, the high quality images captured by the underwater camera system 106 provide the ability to determine other analytic values for management of the fish pen 104.

[0057] From a start block, method 700 proceeds to block 702, where image analysis engine 220 of computing device 206 of underwater camera system 106 detects fish in images captured by stereoscopic camera 202 of underwater camera system 106. In some embodiments, image analysis engine 220 may use machine learning models stored in machine learning model data store 218, including, but not limited to, convolutional neural networks, to detect fish in the images. As described above, images captured by stereoscopic camera 202 may include both color and depth information for each pixel. In some embodiments, method 700 may use techniques such as procedure 400 to determine whether at least one image is suitable for analysis, although such steps are not shown in FIG. 7 .

[0058] At block 704, the image analysis engine 220 determines the distance between the fish and the variable focus lens camera 204 of the underwater camera system 106 based on the depth information in the images. In some embodiments, the image analysis engine 220 knows the relative position of the stereoscopic camera 202 and the variable focus lens camera 204 and uses this relative position along with the depth information in the images (representing the distance between the stereoscopic camera 202 and the fish) to determine the distance between the fish and the variable focus lens camera 204. In some embodiments, the variable focus lens camera 204 is positioned in a manner including, but not limited to, between the lenses of the stereoscopic camera 202, such that the depth information in the images can be used as the distance between the variable focus lens camera 204 and the fish without further processing.

[0059] In optional block 706, the image analysis engine 220 predicts the future location of the fish based on the fish's current location and the fish's movement behavior. In some embodiments, the image analysis engine 220 may use indications of the fish's movement and / or orientation in either the image of the fish or multiple consecutive images to determine the fish's movement and / or orientation. This block is shown and described as optional because, in some embodiments, a single image of a slowly moving fish may be analyzed and the predicted movement of the fish may not be important for the analysis.

[0060] At block 708, the image analysis engine 220 adjusts the focus of the variable focus lens camera 204. In some embodiments, the image analysis engine 220 may adjust the focus of the variable focus lens camera 204 to match the depth information (or processed depth information) from block 704. In some embodiments, the image analysis engine 220 may adjust the focus of the variable focus lens camera 204 to follow the predicted movement of the fish. In some such embodiments, the image analysis engine 220 may continuously measure depth information from the stereoscopic camera 202 and continuously adjust the focus of the variable focus lens camera 204 based on the predicted movement of the fish. In some embodiments, the predicted movement of the fish may include one or more of a vector representing the movement of the fish, an orientation of the fish, and a body movement of the fish.

[0061] In block 710, the image analysis engine 220 captures an in-focus image of the fish using the variable focus lens camera 204, and in block 712, the image analysis engine 220 determines at least one fish monitoring analytic value based at least in part on the in-focus image. Because the focus of the variable focus lens camera 204 is adjusted to capture an in-focus image of the fish, a high-resolution view of the fish can be obtained from which a detailed analysis of the fish's condition can be determined.

[0062] An example of a fish monitoring analytic that can be determined based on the focused images and predicted movement is the presence or absence of skin disease, sea lice, or amoebic gill disease. By focusing the variable focus lens camera 204 on the fish, high resolution images of the fish's skin can be captured, so that these conditions can be identified in the fish from the focused images using any suitable technique, including, but not limited to, machine learning models trained to detect signs of sea lice, skin disease, or amoebic gill disease.

[0063] The method 700 then proceeds to an end block and ends. While FIG. 7 shows the method 700 as ending at this point, in some embodiments, the fish monitoring analytics can be used to control the operation of the fish pen 104. For example, in some embodiments, the fish monitoring analytics can be transmitted to the aquaculture management computing device 102 for presentation to a user so that the user can take action. As another example, in some embodiments, the fish monitoring analytics can be used by the aquaculture management computing device 102 to automatically control the function of the fish pen 104. For example, if sea lice are detected, treatment for sea lice can be provided, such as dispensing H2O2, providing an additive to feed to prevent sea lice, or introducing other fish that eat sea lice.

[0064] 8 is a flowchart illustrating a non-limiting exemplary embodiment of a method for determining at least one fish-monitoring analytic value according to various aspects of the present disclosure. Some fish-monitoring analytic values can be determined based on the movement behavior of the fish even if no focused images are acquired.

[0065] From a start block, method 800 proceeds to block 802, where image analysis engine 220 of computing device 206 of underwater camera system 106 detects fish in at least one image captured by a camera of underwater camera system 106. In some embodiments, image analysis engine 220 may use a machine learning model stored in machine learning model data store 218, including, but not limited to, a convolutional neural network, to detect fish in the images. Image analysis engine 220 may use one or more images captured by stereo camera 202 or variable focus lens camera 204. In some embodiments, method 800 may use techniques such as procedure 400 to determine whether at least one image is suitable for analysis, although such steps are not shown in FIG. 8 .

[0066] At block 804, the image analysis engine 220 determines the movement behavior of the fish. The movement behavior may include the vector and / or orientation of the fish. In some embodiments, the image analysis engine 220 may use indications of the movement and / or orientation of the fish in either the image of the fish or multiple sequential images to determine the movement and / or orientation of the fish.

[0067] At block 806, the image analysis engine 220 determines at least one fish-monitoring analytic value based at least in part on the movement behavior of the fish.

[0068] An example of a fish monitoring analytic that can be determined based on movement behavior is the presence or absence of a trematode infection. In some fish species, a trematode infection can cause the fish to turn while swimming and rub against structures in the fish enclosure 104, such as netting or rope, causing the fish to swim upside down. Detecting the fish turning backwards and / or movements that bring the fish into contact with underwater structures can indicate the presence of a trematode infection.

[0069] Another example of a fish monitoring analytic that can be determined based on movement behavior is stress in fish species with swim bladders, including, but not limited to, trout and steelhead. A shrunken swim bladder can occur when a fish remains submerged and is unable to reach the surface to refill its swim bladder with air to control buoyancy. Movement behavior can indicate the presence of such stress when predicted movements show the fish swimming sideways and / or showing signs of lethargy as buoyancy is no longer compensated by the swim bladder.

[0070] Yet another example of a fish monitoring analytic that can be determined based on movement behavior is fish satiation. It has been observed that some fish species are known to change their swimming style and schooling patterns when they become satiated. Therefore, to determine satiation as a fish monitoring analytic, movement behavior can be compared to known swimming style and schooling patterns that indicate satiation.

[0071] The method 800 then proceeds to an end block and ends. While FIG. 8 shows the method 800 as ending at this point, in some embodiments, the fish monitoring analytics can be used to control the operation of the fish pen 104. For example, in some embodiments, the fish monitoring analytics can be transmitted to the aquaculture management computing device 102 for presentation to a user so that the user can take action. As another example, in some embodiments, the fish monitoring analytics can be used by the aquaculture management computing device 102 to automatically control the function of the fish pen 104. For example, if a fluke infection is detected, an H2O2 dispenser can be activated to flood the fish pen 104 with a dilute concentration of H2O2 for an amount of time effective to address the infection, such as a few seconds or minutes. As another example, if stress associated with swim bladders is detected, a submerged fish pen 104 can be surfaced to allow the fish to refill their swim bladders, or the functionality of artificial air bubbles provided while the fish pen 104 is submerged can be verified or activated. As a final example, the presence or absence of detected satiation can be used to control a food dispenser.

[0072] FIG. 9 is a block diagram illustrating aspects of an exemplary computing device 900 suitable for use as a computing device of the present disclosure. While several different types of computing devices are described above, the exemplary computing device 900 illustrates various elements common to many different types of computing devices. While FIG. 9 is described with reference to a computing device implemented as a device on a network, the following description is applicable to servers, personal computers, mobile phones, smartphones, tablet computers, embedded computing devices, and other devices that may be used to implement some of the embodiments of the present disclosure. Some embodiments of the computing device may be implemented in or include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other customized device. Furthermore, those skilled in the art and others will recognize that the computing device 900 may be any one of any number of devices currently available or yet to be developed.

[0073] In its most basic configuration, computing device 900 includes at least one processor 902 and a system memory 904 connected by a communications bus 906. Depending on the exact configuration and type of device, the system memory 904 may be volatile or non-volatile memory, such as read-only memory (“ROM”), random-access memory (“RAM”), EEPROM, flash memory, or similar memory technologies. Those skilled in the art and others will recognize that the system memory 904 typically stores data and / or program modules that are immediately accessible to and / or presently being operated on by the processor 902. In this regard, the processor 902 can act as the computational center of the computing device 900 by supporting the execution of instructions.

[0074] As further shown in FIG. 9 , computing device 900 may include a network interface 910 comprising one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize network interface 910 to perform communications using a common network protocol. Network interface 910 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as Wi-Fi, 2G, 3G, LTE, WiMAX, Bluetooth, Bluetooth low energy, etc. As will be appreciated by those skilled in the art, network interface 910 shown in FIG. 9 may represent one or more wireless or physical communication interfaces described and illustrated above with respect to particular components of computing device 900.

[0075] In the exemplary embodiment shown in Figure 9, computing device 900 also includes storage medium 908. However, services may be accessed using computing devices that do not include means for persisting data to local storage medium. Accordingly, storage medium 908 shown in Figure 9 is represented by a dashed line to indicate that storage medium 908 is optional. In any event, storage medium 908 may be volatile or non-volatile, removable or non-removable, and may be implemented using any technology capable of storing information, such as, but not limited to, a hard drive, solid-state drive, CD-ROM, DVD, or other disk storage, magnetic cassette, magnetic tape, magnetic disk storage, etc.

[0076] Suitable implementations of a computing device including a processor 902, system memory 904, communication bus 906, storage medium 908, and network interface 910 are known and commercially available. For ease of explanation and because they are not important to an understanding of the claimed subject matter, FIG. 9 does not show some of the typical components of many computing devices. In this regard, computing device 900 may include input devices such as a keyboard, keypad, mouse, microphone, touch input device, touch screen, tablet, etc. Such input devices may be coupled to computing device 900 by wired or wireless connections, including RF, infrared, serial, parallel, Bluetooth, Bluetooth low energy, USB, or other suitable connection protocols using wireless or physical connections. Similarly, computing device 900 may also include output devices such as a display, speakers, printer, etc. These devices are well known in the art and therefore will not be further shown or described herein.

Claims

1. 1. An underwater camera system for monitoring at least one fish in a fish pen, comprising: A stereoscopic camera and A variable focus lens camera; a computing device communicatively coupled to the variable focus lens camera and the stereoscopic camera, Sequentially detecting fish in a plurality of sequential images from the stereoscopic camera; determining a vector representing the movement behavior of the fish from the plurality of consecutive images; continuously adjusting the focus of the variable focus lens camera based on the plurality of successive images and the vector indicative of the motion behavior; capturing at least one focused image of the fish with the variable focus lens camera after continuously adjusting the focus of the variable focus lens camera; determining at least one fish-monitoring analytic value based at least in part on the at least one focused image of the fish; a computing device configured to Underwater camera system including.

2. The underwater camera system of claim 7 , further comprising a housing, wherein the stereoscopic camera, the variable focus lens camera, and the computing device are disposed within or attached to the housing.

3. determining the at least one fish monitoring analytic value comprises: Determining the presence or absence of fluke infection; Determining the presence or absence of stress in fish species having swim bladders; Determining whether satiation occurs and 8. The underwater camera system of claim 7, comprising at least one of:

4. The computing device may be further configured to determine whether the fish is measurable prior to determining the at least one fish-monitoring analytic value, and determining whether the fish is measurable includes: detecting blur in at least one image of the plurality of sequential images; applying a profilometric analysis to at least one image of the plurality of sequential images; and detecting an occlusion of the fish in at least one image of the plurality of consecutive images; 8. The underwater camera system of claim 7, comprising at least one of:

5. continuously adjusting the focus of the variable focus lens camera based on the plurality of successive images from the stereoscopic camera; determining a distance from the variable focus lens camera to the fish based on depth information in the plurality of sequential images from the stereo camera; continuously adjusting the focus of the variable focus lens camera based on the distance; 8. The underwater camera system of claim 7, comprising:

6. continuously adjusting the focus of the variable focus lens camera based on the plurality of successive images from the stereoscopic camera, predicting a future position of the fish based on the current position of the fish and the vector indicating the movement behavior of the fish; continuously adjusting the focus of the variable focus lens camera based on the predicted future position; 8. The underwater camera system of claim 7, comprising:

7. determining the at least one fish-monitoring analytic based at least in part on the at least one focused image; Detecting a skin condition of the fish; detecting signs of amebic gill disease in the fish; detecting sea lice on said fish; 8. The underwater camera system of claim 7, comprising at least one of:

8. determining the at least one fish-monitoring analytic based at least in part on the at least one focused image; determining the at least one fish-monitoring analytic value based on the movement behavior; and 8. The underwater camera system of claim 7, comprising:

Images