Method and system for fish ectoparasite monitoring in aquaculture

Abstract

A method for fish ectoparasite monitoring in aquaculture, comprising the steps of: - submerging a camera (52) into a sea pen containing fish, said camera having a field of view; - capturing images of the fish with the camera (52); and - identifying fish ectoparasites on the fish by analyzing the captured images, characterized in that a target area within the field of view of the camera (52) is illuminated from above and below with light having different intensities and / or spectral composition.

Description

[0001] This case is a divisional application of patent application number 201880082865.3, filed on 19 December 2018, entitled “Method and system for fish ectoparasite monitoring in aquaculture”, of the same applicant, the whole content of the parent application being incorporated herein by reference. TECHNICAL FIELD

[0002] The present invention relates to a method for fish ectoparasite (such as sea lice) monitoring in aquaculture, comprising the steps of:

[0003] - submerging a camera into a sea pen containing fish, said camera having a field of view;

[0004] - capturing images of said fish with said camera; and

[0005] - identifying fish ectoparasites such as sea lice on said fish by analyzing the captured images.

[0006] In the present specification, the term “monitoring” refers to any activity aimed at providing an empirical basis for a decision whether a given fish population is infested with ectoparasites. The term monitoring can also include ways of determining the extent to which fish are infested with ectoparasites. Although monitoring can be combined with measures to destroy or kill the parasites, the term monitoring itself does not include such measures. BACKGROUND

[0007] Like humans and other animals, fish also suffer from diseases and parasites. Parasites can be internal (endoparasites) or external (ectoparasites). Fish gills are a preferred habitat for many fish ectoparasites, which attach to the gills but live outside of them. Most commonly, monogenean subclass and certain groups of parasitic copepods, which can be extremely numerous, are found. Other fish ectoparasites found on gills are leeches, and in seawater, the larvae of isopod crustaceans such as gnathiid. Fish parasites of isopods are mostly external and feed on blood. Larvae of the Gnathiidae family and adult cymothoidids have a piercing and sucking mouth organ, and clawed limbs adapted to cling to their host. Cymothoa exigua is a parasite of various marine fish. It causes the fish’s tongue to atrophy, and is believed to be the first example of a parasite found in animals that functionally replaces a host structure. Among the most common fish ectoparasites are the so-called sea lice.

[0008] Sea lice are small parasitic crustaceans (Copepoda) that feed on the mucus, tissue and blood of marine fish. Sea lice (plural sea lice) are members of the order Siphonostomatoida - the Caligidae. There are approximately 559 species in 37 genera, including approximately 162 species of Lepeophtheirus and 268 species of Caligus. While sea lice are present in wild salmon populations, sea lice infestations in farmed salmon populations present a particular challenge. A number of anti-parasitic drugs have been developed for control purposes. L. salmonis is the main sea louse of concern in Norway. Caligus rogercresseyi has become the main parasite of concern for salmon farms in Chile.

[0009] Sea lice have both a free-swimming (planktonic) and a parasitic life stage. All stages are separated by molting. The rate of development of L. salmonis from egg to adult varies from 17 to 72 days depending on temperature. The egg hatches into nauplius I, which molts to the second nauplius stage; neither nauplius stage feeds, relying on energy reserves from the yolk, and are adapted for swimming. The copepodid stage is the infective stage, and it searches for a suitable host, possibly through chemical and mechanical sensory cues.

[0010] Once attached to a host, the copepodid stage begins to feed and develop into the first chalimus stage. The copepodid and chalimus stages have a developed gastrointestinal tract and feed on host mucus and tissue within their attachment range. Pre-adult and adult sea lice (particularly pregnant females) are aggressive feeders, in some cases feeding on blood in addition to tissue and mucus.

[0011] The time and expense associated with mitigation efforts, as well as fish mortality, add approximately 0.2 Euro / kg to the cost of fish production. Fish external parasites such as sea lice are therefore a major concern for contemporary salmon farmers, who require significant resources for avoiding infestations and complying with government regulations aimed at avoiding broader ecological impacts.

[0012] Both efficient mitigation (e.g. assessing the need and timing of vaccination or chemotherapy) and regulatory compliance rely on accurate quantification of fish ectoparasites (e.g. sea lice populations in a single farming operation). Currently, counting fish ectoparasites (e.g. sea lice) is completely manual, and thus a very time consuming process. For example, in Norway, counts must be performed and reported weekly, incurring direct costs of 24 million USD per year for this alone. Also problematic is the questionable validity of statistics based on manual counts when counts of fish ectoparasites (e.g. adult female lice) on a sample of 10 to 20 fish under sedation are extrapolated to determine appropriate treatment of a population of over 50,000 fish. Thus, both over-treatment and under-treatment are common.

[0013] WO2017 / 068127A1 describes a system of the type indicated in the preamble of claim 1, with the objective of being able to automatically and accurately detect and count fish ectoparasites, such as sea lice, within a population of fish.

[0014] Any such system based on optical imaging must overcome a number of significant challenges related to the marine environment and animal behavior.

[0015] - Optical distortions caused by density gradients. Turbulent mixing of warm and cold water, or, in particular, salt and fresh water (e.g. within a fjord) creates small scale density variations, leading to optical distortions. The impact on imaging of objects smaller than 1-3 mm (e.g. sea lice larvae) is particularly severe.

[0016] - Fish aversion to unfamiliar light sources. Fish can exhibit a fearful reaction or more general aversion to light sources of unfamiliar location, intensity or spectrum. Distortion of the fish population around such light sources will typically increase the distance from the typical imager to the fish, reducing the effective acuity of the imaging system. The cited document addresses this problem by providing a guidance system for guiding the fish along a desired imaging trajectory.

[0017] - Focus tracking in highly dynamic marine environments. Commercially available focus tracking systems perform poorly in highly dynamic scenes where there are a large number of fast moving, seemingly real focus targets (i.e. a school of swimming fish) within the field of view simultaneously.

[0018] It is an object of the present invention to provide a system and method that addresses these challenges and provides accurate automatic counting, to reduce the amount of manual labor related to fish ectoparasites (e.g. sea lice counting) and to enable more efficient prediction and prevention of harmful infestations. SUMMARY

[0019] To this end, the method according to the invention is characterized in that the target area within the field of view of the camera is illuminated from above and below with light having different intensities and / or spectral composition.

[0020] The present invention takes into account the fact that fish generally have a relatively dark color on their dorsal side and a relatively light color on their ventral side. By specifically adapting the intensity and / or spectral composition of the illumination light to these different colors, optimal contrast conditions can be provided for the detection of fish ectoparasites, such as sea lice, whether they are located on the dorsal side or on the ventral side of the fish.

[0021] More specific optional features of the present invention are indicated in the dependent claims.

[0022] Preferably, the system is capable of detecting and classifying fish ectoparasites such as sea lice of both sexes in a wide variety of sessile, motile and spawning life stages (e.g. larva, pre-adult, adult male, adult egg-bearing female and adult non-egg-bearing female).

[0023] Moreover, the system can form the basis of an integrated decision support platform to improve operational performance, animal health and sustainability of marine-based aquaculture. BRIEF DESCRIPTION OF DRAWINGS

[0024] Embodiment examples will now be described in conjunction with the accompanying drawings in which:

[0025] Figure 1 A side view of a camera and lighting rig according to a preferred embodiment of the present invention is shown;

[0026] Figure 2 is a view of a sea pen with a camera and lighting rig according to Figure 1 suspended therefrom;

[0027] Figure 3 A front view of the camera and lighting rig is shown;

[0028] Figure 4 A side view of the angular field of view of a range-finding detector mounted on the rig is shown, as well as a side view of the lighting rig;

[0029] Figure 5 and Figure 6 is a graph illustrating the detection results of the range-finding detector;

[0030] Figures 7-10 An image frame is shown, which illustrates multiple steps of an image capture and analysis procedure;

[0031] Figure 11 A flow chart is shown, which details the process of annotating images as well as fish ectoparasite detector training, validation and testing within an electronic image processing system (machine vision system) according to an embodiment of the present invention; and

[0032] Figure 12A flowchart is shown which details the operation of the fish external parasite detector in inference mode. DETAILED DESCRIPTION

[0033] Image capture system

[0034] As Figure 1 shown, the image capture system comprises a camera and lighting rig 10 and a camera and lighting control system 12 which enables high quality images of fish to be taken automatically.

[0035] The camera and lighting rig 10 comprises a vertical support member 14, an upper boom 16, a lower boom 18, a camera housing 20, an upper lighting array 22 and a lower lighting array 24. The camera housing 20 is connected to the vertical support member 14 and is preferably height adjustable. The vertical positioning of the camera is preferably such that the field of view of the camera is at least partially (preferably mostly or wholly) covered by the illumination cones of the upper and lower lighting arrays 22, 24. Also, preferably there is a substantial angular offset between the centre line of the camera field of view and the centre line of the illumination cones. This minimises the amount of light backscattered to the camera (by particles in the water) and thus (relatively) maximises the amount of light returned from the fish tissue. In the illustrated arrangement, the camera can be mounted at a height of between ¼ and ¾ of the length of the vertical support member, as measured from the lower end of the support member 14.

[0036] The upper boom 16 and lower boom 18 are coupled to the vertical support member 14 at elbow joints 26 and 28 respectively, which allow angular articulation of the upper and lower booms relative to the vertical support member. The upper lighting array 22 and lower lighting array 24 are coupled to the upper and lower booms at pivotable joints 30 and 32 respectively, which allow angular articulation of the upper and lower lighting arrays relative to the upper and lower booms.

[0037] In the illustrated example, a suspension cord 34 constitutes a two-line suspension for the camera and lighting rig 10. The suspension cord allows control of the rig's posture in azimuth, and can be connected to a bracket 36 at different positions, thereby maintaining the rig's balance for a given configuration of the booms 16 and 18. This enables fine adjustment of the camera and lighting rig's orientation (i.e. pitch angle) since the camera and lighting rig's centre of mass is below the connection point.

[0038] Preferably, a wiring conduit 38 carries all the data and power required between the camera and lighting rig and the camera and lighting control system 12 for the upper and lower lighting arrays and the camera housing.

[0039] Figure 2A diagram showing the camera and lighting rig 10 submerged in a sea pen 40. The example sea pen shown is surrounded by a dock 42, from which vertical support members 44 extend upwards. Taut cables 46 span between the support members. The suspension line 34 can be connected to the taut cables 46 to allow insertion and removal of the camera and lighting rig 10 into and out of the sea pen, as well as control of the horizontal position of the rig relative to the dock 42.

[0040] It should be noted, however, that the sea pen can also have a shape other than Figure 2 the shape shown.

[0041] The extension of the support cables and lines also allows adjustment of the depth of the camera and lighting rig below the water surface. Preferably, the camera and lighting rig is placed at a depth that positions the camera housing 20 below the surface mixed layer where the turbulent mixing of warm and cold water, or salt and fresh water, is most pronounced. This further reduces optical distortion associated with the density gradient. The required depth varies based on location and season, but a depth of 2-3 m is generally preferred.

[0042] As Figure 3 shown, the upper lighting array 22 and the lower lighting array 24 comprise horizontal members 48 that support one or more lighting units 50 within the lighting array along their length. In Figure 3 the embodiment shown, the upper lighting array and the lower lighting array each comprise two lighting units 50, however a different number of lighting units can be used. The horizontal members 48 are coupled to the upper boom and the lower boom at pivotable joints 30, 32.

[0043] The elbow joints 26, 28 between the vertical support members 14 and the upper boom 16 and the lower boom 18, and the pivotable joints 30, 32 between the upper boom and the lower boom and the horizontal members 48 collectively allow independent adjustment of:

[0044] - the horizontal offset between the camera housing 20 and the upper lighting array 22,

[0045] - the horizontal offset between the camera housing 20 and the lower lighting array 24,

[0046] - the angular orientation of the lighting units 50 within the upper lighting array 22, and

[0047] - the angular orientation of the lighting units 50 within the lower lighting array 24.

[0048] Generally, the upper and lower lighting arrays are positioned relative to the camera housing to provide sufficient illumination within the target area where the fish will be imaged for fish external parasite (e.g., sea lice) detection. The longitudinal, vertical design and configuration of the camera and lighting assembly 10 maximizes the likelihood that the fish, which exhibits a phobia of long, horizontally oriented objects, will swim in close proximity to the camera housing. Moreover, the separate and independently adjustable upper and lower lighting arrays allow for lighting schemes specifically designed to address the unique lighting challenges of fish, as discussed in greater detail below.

[0049] In Figure 3 The camera housing 20, shown in elevation view in FIG. 1, includes a camera 52, a range detection detector 54 (e.g., a light-based time-of-flight detection and ranging unit), and a pose sensing unit 56, which includes, for example, a magnetometer and an inertial measurement unit (IMU) or other known pose sensing system.

[0050] The camera is preferably a commercially available digital camera with a high sensitivity, low noise sensor capable of capturing clear images of fast moving fish under relatively low illumination. In a preferred embodiment of the present application, a Raytrix C42i camera is used, which provides a horizontal field of view of approximately 60° and a vertical field of view of approximately 45°. Of course, any other camera with similar performance (including electronically controllable focus) can be used as an alternative.

[0051] The range detection detector 54 is used to detect the range and bearing of a fish swimming within the field of view of the camera 52. The detector includes an emission optic 58 and a reception optic 60. The emission optic 58 produces a sector of light that is directed in the vertical direction, but preferably collimated in the horizontal direction. That is, the sector diverges in pitch parallel to the vertical support member, but relatively little in yaw perpendicular to the vertical support member.

[0052] The reception optic 60 includes an array of light detector elements, each of which detects light incident from within an acceptance angle spanning at least a portion of the vertical field of view of the camera. The angles of adjacent detector elements pitch abut one another, collectively establishing an acceptance fan that completely covers the vertical field of view. This orientation and configuration of the emission and reception optics is optimized to detect and locate a fish body (which is generally high aspect ratio) swimming in parallel to the horizontal water surface.

[0053] Preferably, the range detection detector 54 operates on a wavelength of light that provides effective transmission in water. For example, blue or green light can be used to provide effective transmission in seawater. In a preferred embodiment of the present application, the range detection detector is a The detectors, such as LeddarTech M16, emit and receive light at 465 nm. Of course, the present application is not limited to this implementation of range-finding detectors.

[0054] Also in preferred embodiments of the present application, the illumination sector produced by the emitting optics has an approximately 45° elevation divergence, effectively spanning the vertical field of view of the camera, and an approximately 7.5° yaw divergence. The receiving optics 60 include an array of 16 detector elements, each having a field of view spanning approximately 3° in elevation and approximately 7.5° in yaw. Of course, the number of detector elements can be less than or greater than 16, but is preferably not less than 4. Preferably, both the illumination sector and the acceptance sector are horizontally centered within the camera field of view, ensuring that a detected fish can be fully captured by the camera.

[0055] Systems with two or more range-finding detectors can also be envisioned. For example, a sector can be positioned "upstream" of the centerline (defined by the prevailing direction of fish movement) to provide a "high level warning" of fish entering the pen. Similarly, a unit can be placed downstream to confirm fish leaving the pen.

[0056] The IMU in the attitude sensing unit 56 includes an accelerometer and a gyroscope, for example, similar to those found in commercially available smartphones. In preferred embodiments of the present application, a magnetometer and the IMU are co-located on a single printed circuit board within the camera housing 20. Together, the IMU and the magnetometer measure the orientation of the camera housing (and thus the images acquired by the camera) relative to the water surface and the sea pen. Because fish typically swim parallel to the water surface and along the edge of the sea pen, this information can be used to inform the machine vision system of the expected orientation of fish in the captured images.

[0057] The upper and lower illumination arrays 22 and 24 can include one or more lamps of various types (e.g., incandescent, gas discharge, or LED) that emit light of any number of wavelengths. Preferably, the particular type of lamp is chosen to provide sufficient color information (i.e., a broad enough emission spectrum) to fish external parasites (e.g., sea lice) to contrast sufficiently with the fish tissue. Additionally, the type and intensity of the lamps within the upper and lower illumination arrays are preferably chosen to produce a relatively uniform intensity of light reflected to the camera, despite the typically, and notably, countershaded body of the fish.

[0058] In the implementation presented here, the upper lighting array 22 comprises a pair of xenon flash tubes. The lower lighting array 24 comprises a pair of LED lights, each comprising a chip with 128 white LED dies. This hybrid lighting system provides a greater range of lighting intensities than can be achieved with a single lighting type. Specifically, the flash tubes are synchronized with the operation of the camera shutter to provide brief but intense illumination (approximately 3400 lx) from above the fish. This ensures sufficient light from the fish's typically dark, highly absorbent upper surface to be reflected to the camera. (This requires greater intensity light than the LED lights of the lower lighting array can deliver.) Correspondingly, the LED lights provide sufficient lighting intensity for the fish's typically bright, highly reflective lower surface. (This requires less intensity than the xenon flash tubes of the upper lighting array can provide.) The resulting uniform bright light reflected from the fish allows the camera to operate at lower sensitivity (e.g., below ISO 3200) to provide low-noise images to the machine vision system. Finally, both the xenon flash tubes and the LED lights provide a sufficiently wide spectrum to allow differentiation between fish tissue and external parasites of the fish such as sea lice.

[0059] As noted above, the upper lighting array 22 and the lower lighting array 24 are positioned to provide desired illumination in the entire target region. The target region is characterized by a vertical field of view of the camera and a near boundary and a far boundary along the axis of the camera. The distance from the camera to the near boundary is the farther of (a) the closest reachable focal length of the camera, and (b) the distance that a typical fish spans the entire horizontal angle of view of the camera. The distance from the camera to the far boundary is the distance at which the angular resolution of the camera can no longer resolve the smallest external parasites of the fish (e.g., sea lice) that must be detected. The near boundary "a" and the far boundary "b" are illustrated in Figure 4

[0060] Each light within the upper and lower lighting arrays provides a generally axis-symmetric illumination pattern. Because there are multiple lights within each array along the length of the horizontal members, the illumination pattern can be effectively characterized by an angular span in the elevation plane. The length of the vertical support member 14, the angular position of the upper boom 16 and the lower boom 18, and the angular orientation of the upper lighting array 22 and the lower lighting array 24 are preferably adjusted so that the angular span of the upper and lower lighting arrays effectively covers the target region. The distance from the camera to the "sweet spot" depends on the size of the fish to be monitored, and can be in the range of 200 mm to 2000 mm, for example. In the case of salmon, for example, a suitable value can be about 500 mm.

[0061] ​In practice, the intensity of the illumination provided by the upper and lower lighting arrays is not perfectly uniform over their angular span. The above approach, however, ensures that an acceptable amount of illumination is provided over the target region. It also creates a "sweet spot" at a short distance outside the near boundary, where the angle of the illumination between the upper and lower lighting arrays and the camera is optimal. This results in the best-illuminated image, thus also providing the best angular resolution achievable by the camera, and being least affected by density gradient distortion.

[0062] An example of a suitable configuration and sizing of the lighting rig will now be explained with reference to Figure 4 the average length of the fish to be monitored is 80 cm. The elbow joints 26, 28 are adjusted so that the respective horizontal offsets H01 and H02 between the camera housing (in particular, the sensor plane within the camera) and the upper and lower lighting arrays are approximately 33 cm and 21 cm, respectively. In units of the average length of the fish, suitable values can be 0.41 ± 0.04 and 0.26 ± 0.03, respectively. These angular orientations of the upper and lower booms 16, 18 also provide vertical offsets V01 = 71 cm and V02 = 52 cm between the camera housing and the upper and lower lighting arrays, respectively. In units of the average length of the fish, suitable values can be 0.89 ± 0.09 and 0.65 ± 0.07, respectively. The pivotable joints 30, 32 are adjusted so that the upper lighting array is tilted 63° (± 10%) of an angle a1 below the horizontal, and the lower lighting array is tilted 47° (± 10%) of an angle a2 above the horizontal. As mentioned above, in the preferred embodiment, the vertical field of view y of the camera is approximately 45° (± 10%). The angular spans of the upper and lower lighting arrays are approximately d1 = 62° (± 10%) and d2 = 60° (± 10%), respectively. (Those skilled in the art will appreciate that these angular span measurements are approximate, since the "drop-off" in light intensity at the edges of the angular span is gradual.) The field of view and the angular spans combine to produce a target region having a near boundary NB of approximately 30 cm from the camera housing, and a far boundary FB of approximately 110 cm from the camera housing. This results in a target region TR having a depth of approximately 80 cm, with a sweet spot SP of approximately 40 cm in depth (approximately corresponding to the near half of the target region). The width and height of the target region will be approximately 80 cm.

[0063] A wide variety of other camera and lighting geometries can be used without departing from the scope of the present invention. In particular, the camera and lighting rig can be constructed and positioned in other than Figure 1other than the vertical direction. For example, the camera and lighting rig can be oriented horizontally parallel to the water surface. The camera and lighting rig can also be configured to hold one or more cameras in a fixed position (relative to the target area) other than the position shown. In addition, some embodiments of the present invention can integrate multiple cameras and lighting rigs, for example two cameras and lighting rigs positioned symmetrically in front of and behind the target area, enabling simultaneous capture of images of both sides of a single fish. Figure 1

[0064] Camera and lighting control system

[0065] The camera and lighting control system 12 controls the operation of the image capture system. The camera and lighting control system:

[0066] - receives and analyzes data from the ranging detector 54 to determine the appropriate camera focal length,

[0067] - controls the camera focal length and shutter,

[0068] - controls the timing of the lighting of the upper lighting array 22 and the lower lighting array 24 relative to the shutter of the camera 52, and

[0069] - receives, analyzes, and stores image data and image metadata, including ranging detector, magnetometer, and IMU measurements.

[0070] In the present embodiment, the camera and lighting control system 12 includes a computer 62 and a power control unit 64, which reside at a dry location (e.g., the dock 42) physically proximate to the camera and lighting rig 10. In an alternative embodiment, at least a portion of the camera and lighting control system functionality provided by the computer is performed by an embedded system below the water surface (e.g., mounted to the vertical support member 14 or integrated within the camera housing). Generally, the computer 62 includes device drivers for each sensor within the camera and lighting rig 10. In particular, the computer includes device drivers for the camera 52, the ranging detector 54, and the magnetometer and IMU of the pose sensing unit 56. The device drivers allow the computer to acquire measurement data from the relevant sensors and send control data to the relevant sensors. In the preferred embodiment, the measurement data and control data are passed as messages in a Robot Operating System (ROS) between the devices and processes running on the computer. Data from the sensors (including the ranging detector) arrives at a frequency of 10 Hz, while measurements from the magnetometer and IMU arrive at a frequency of 100 Hz. Each message is logged to a disk on the computer.

[0071] ​The computer 62 provides control signals to the power control unit 64 and optionally receives diagnostic data from the power control unit. The power control unit provides power to the upper lighting array 22 and the lower lighting array 24 through the cable 38. In the preferred embodiment, the power control unit receives 220V AC power, which can be passed directly to a charger for the capacitor bank used for the xenon flash tubes (when triggered) within the upper lighting array 22. The power control unit passes the power to an underwater junction box (not shown), which converts the AC power to DC power (e.g., 36V or 72V) for the LED lights within the lower lighting array 24.

[0072] The focal point calculation process executed by the computer 62 continuously monitors the ranging data to detect the presence of fish within the target area and to determine their extent. For each detector element, the ranging data consists of one or more distances from which light is reflected back to the detector element within its acceptance angle within the acceptance sector.

[0073] Figure 4 A side view of the angular field of view 66 of a detector element within the receiving optics 60 within the ranging acceptance sector 68 is shown. As described above, the acceptance angles of adjacent detectors are tilted next to each other to establish the acceptance sector. Figure 4 An array of 16 detectors is shown, each having a field of view spanning approximately 3° of tilt.

[0074] Figure 4 A side view of the average tilt angle of the detectors within the ranging acceptance sector 68 is shown. Each average tilt angle is illustrated by a center line 70 bisecting the field of view 66 of the corresponding detector. The average tilt angle is the angle between the bisecting center line 70 and the center line of the acceptance sector 68 as a whole, which is generally parallel to the optical axis of the camera 52.

[0075] Figure 4 A side view of the distances and average tilt angles of several detector elements blocked by fish 72, 74 within the ranging acceptance sector 68 is also shown. In general, the focal point calculation process detects a fish when several adjacent detector elements report similar distances. In the preferred embodiment of the invention, a fish is detected when M or more adjacent detector elements report similar distances d i The number M can range from 1 to ½ of the total number of detectors (i.e., 8 in this example). Specifically, the focal point calculation process looks for M or more adjacent sets of similar distances d i The number M can range from 1 to ½ of the total number of detectors (i.e., 8 in this example). Specifically, the focal point calculation process looks for M or more adjacent sets of similar distances d i The number M can range from 1 to ½ of the total number of detectors (i.e., 8 in this example). Specifically, the focal point calculation process looks for M or more adjacent sets of similar distances d i)] < W. M and W are parameters that can be adjusted by the operator of the image capture system, where W represents the maximum allowed thickness, approximately corresponding to half the thickness of the largest fish to be detected. Depending on the size or age of the fish, the parameters M and W are optimized for each system or pen. For each such detection, the focus calculation will calculate the average distance

[0076] D = (1 / M)∑i M d i

[0077] and the average orientation

[0078] β = (1 / M)∑i M β i

[0079] where β i is the average elevation angle of the respective neighboring detector elements. The focus calculation process then returns the focal distance D f = D*cos β, which represents the distance from the camera along the optical axis of the camera to the recommended focal plane.

[0080] Due to scattering particles in the water or objects (e.g. fish) that only accept a portion of the detector acceptance angle, multiple distances can be reported by a single detector. In actual implementations, in those cases where a single detector reports multiple distances, the focus calculation uses the farthest distance. This minimizes the number of false detections caused by particles in the water. In cases where multiple distances are actually associated with two fish, one of which only partially blocks the detector acceptance angle, it is likely that the neighboring detector will still successfully detect the partially blocked fish.

[0081] The presence and range information of the fish determined by the focus calculation process can be used to control the image acquisition of the camera. For example, an image can only be captured if a fish is detected within a predetermined distance of the “sweet spot” that provides the best lighting. For example, if the “sweet spot” is located at 500 mm, an image can only be captured if a fish is detected within a distance range of 300 to 1100 mm. Whenever an image is captured, the camera and lighting control system sets the focal distance of the camera to the nearest range value determined by the focus calculation process.

[0082] In a preferred embodiment of the invention, the camera and lighting control system continuously periodically triggers the camera to acquire images, for example at a frequency of 4 Hz or more typically between 2 and 10 Hz. The focus calculation process continuously and periodically (for example at 10 Hz or more typically between 4 and 20 Hz) reports the current focal distance and distance based on the latest fish detection, and the camera and lighting control system sets the focal distance of the camera to the latest available focal distance.

[0083] When the camera shutter opens, the camera sends a synchronization signal to the camera and lighting control system 12, which is then transmitted to the power control unit 64. The power control unit illuminates the upper illumination array 22 and the lower illumination array 24 in sync with the shutter to ensure proper illumination of the captured image. In those embodiments of the invention where the lamps in the upper or lower illumination array cannot maintain a duty cycle equal to that of the camera (e.g., the xenon flash tube in a preferred embodiment of the invention), the power control unit may further include an illumination suppression process that continuously evaluates whether the power control unit should illuminate the upper and lower illumination arrays. In a preferred embodiment of the invention, illumination suppression occurs if (a) the emission history of the xenon flash tube in the upper illumination array is close to its thermal limit, or (b) the focus calculation process has not recently detected a flash and reports an updated range.

[0084] In a preferred embodiment of the invention, the lower-intensity LEDs in the lower illumination array illuminate for the duration of the camera exposure. The exposure length is set to the minimum length required for the LEDs to provide sufficient illumination. The flash length of the xenon flash tubes in the upper illumination array is adjusted to provide balanced illumination taking into account the contrasting brightness of typical fish.

[0085] The illumination provided by the LEDs in the lower illumination array is preferably sufficiently intense to provide a sufficiently short exposure to produce acceptable low motion blur within the captured image of the swimming fish. In a preferred embodiment of the invention, the sensors within the camera (particularly its pixel count), the camera's optics (particularly the angular span of the field of view), and the distance to the target area are selected to ensure that (a) the whole fish can be captured within the camera's field of view, and (b) even sufficient resolution of external parasites on juvenile fish, such as juvenile sea lice. Providing 10 pixels per 2 mm (equivalent to the size of a juvenile sea lice) at a target distance spanning the width of a typical adult fish in the camera's 60° horizontal field of view requires 7.6 × 10 pixels per pixel. -3 The angular pixel spacing is °. For a fish swimming at a typical speed of 0.2 m / sec, the angular pixel spacing is less than 0.6 × 10⁻⁶. -3 A shutter speed of s ensures subpixel motion blur. To provide sufficiently low-noise images to the machine vision system, a sensor gain of less than ISO 3200 is preferred. This, in turn, requires illumination of approximately 3000 lux across the entire target area.

[0086] Figure 5 This explains that in Figure 4 The results were obtained using the range detector 54 in the depicted scenario. The results shown are from a detector element with a field of view of the centerline ranging from +6° to -15°. Figure 5Each black dot in the plot represents a detection event in which reflected light has been received by the associated detector element. The position of the dot in the d-axis direction represents the distance of the detected object, as calculated from the time of flight of the light signal from the emitting optics 58 to the object and back to the receiving optics 60.

[0087] As previously explained, the fish 72 and 74 are represented by detections at approximately the same distances dl and d2, respectively, for a number of adjacent detectors. For each individual detector, the distance of the fish is the largest of the distances measured by that detector. The dots at smaller distances represent noise caused by small particulate matter in the acceptance sector.

[0088] In the case illustrated in Figure 4 and Figure 5 the fish 74 is partially obscured by the fish 72, so that an image of the entire profile of the fish can only be obtained for the fish 72 at the smaller distance dl. The focal length of the camera will therefore be adjusted to this distance dl.

[0089] Figure 6 is a time plot showing the detections at the distance dl as a function of time t. It can be seen that the detections obtained from the fish 72 (for angles β in the range -3° to -15°) are stable over a long period of time corresponding to the time it takes for the fish to swim through the acceptance sector 68. It is therefore also possible to filter out noise by requiring the detections to be stable over a certain minimum time interval, or equivalently, by integrating the signal received from each detector element over a certain time and then thresholding the result of the integration.

[0090] In principle, the detection history of the type illustrated in Figure 6 may also be used to optimize the time interval in which the camera 52 takes a series of images, so as to ensure that on the one hand the number of images does not become unreasonably large, and on the other hand that the series of images includes at least one image in which the entire fish is within the field of view of the camera. For example, as Figure 6 illustrated, a timer can be started at a time tl when a certain number of adjacent detector elements (three) detect an object which can be a fish. The camera can then be triggered at a time t2 with a certain delay to start taking a series of images, and the series can be stopped at the latest at a time t3 when the detector elements indicate that the tail end of the fish has left the acceptance sector.

[0091] Figure 7 is shown when the nose of the fish 72 has just crossed the acceptance sector 68, Figure 6 the field of view 76 of the camera at the time tl in

[0092] Figure 8 is shown when the entire profile of the fish 72 is within the field of view 76 of the camera 52 at a time t2 later than Figure 6the image captured at a later time t2. At that moment, it can be inferred from the detection results of the detector elements at β = -3° to -15° in Figure 6 that the centerline of the fish will be at β = -9°, as shown. This information can be passed to the image processing system and can help identify the outline of the fish in the captured image. Figure 8

[0093] Returning to Figure 1 , the computer 62 of the camera and lighting control system 12 is connected to the image processing system 78, which accesses the database 80 through the data management system 82.

[0094] Data management system

[0095] The automated system for detecting and counting fish external parasites, such as sea lice, also includes a data management system 82, which includes interfaces that support the acquisition, storage, search, retrieval, and distribution of image data, image metadata, image annotations, and detection data created while the image processing system 78 is running.

[0096] Data storage

[0097] The data management system 82 receives images from the image capture system, for example, in the form of ROS "bags". The data management system unpacks each bag into, for example, JPEG or PNG images and JSON (JavaScript Object Notation) metadata. The JPEG images are each stored within a data store.

[0098] Database

[0099] The JSON metadata unpacked from each ROS bag is stored within the database 80 associated with the data store. Typically, the metadata describes image capture parameters of the associated JPEG or PNG image. For example, the metadata includes an indication of the centroid pixel position within the outline of the fish detected by the LEDDAR unit (e.g., the pixel horizontally centered within the image, the longitudinal centerline of the fish). This pixel position can optionally be used by the image processing system (described in more detail below) to facilitate detection of the fish within the image.

[0100] The database 80 also stores annotation data created during an annotation process used to train the image processing system 78 (described in more detail below). The database also stores information characterizing the location, size, and type of fish and fish external parasites, such as sea lice, detected by the machine vision system. Finally, the database stores authentication credentials that enable users to log into various interfaces (e.g., an annotation interface or an end-user interface) through a verification module.

[0101] Image processing system ​

[0102] In one embodiment, the present application uses an image processing system 78 to perform the task of fish ectoparasite (e.g. sea lice) detection. In a preferred embodiment of the present application, separate neural networks are trained to provide a fish detector 84 and a fish ectoparasite detector 86. The fish detector 84 first detects individual fish within the images acquired by the image capture system. The fish ectoparasite detector 86 then detects individual fish ectoparasites, e.g. sea lice, on the surface of each detected fish, if present. Preferably, the fish ectoparasite detector also classifies the sex and life stage of each detected louse.

[0103] The detectors are trained by a machine learning procedure that ingests a corpus of human-annotated images. The use of neural networks eliminates the need for a definition of characteristics (e.g. extent, shape, brightness, color or texture) of fish or fish ectoparasites (e.g. sea lice) and instead directly exploits the knowledge of the human annotators as encoded within the corpus of annotated images.

[0104] Figure 9 The position and outline of the fish 72 in the field of view 76 is shown, as detected by the fish detector 84. Figure 4 、 7 Another fish 74 shown in Figs. 7 and 8 has been excluded from consideration in this embodiment as it is partially obscured by the fish 72. However in a modified embodiment it is possible to also detect the fish 74 and look for fish ectoparasites (e.g. sea lice) on the skin of the fish 74, if they are visible.

[0105] In one embodiment of the present application, the depth of focus of the camera 52 has been chosen so that a sharp image is obtained for the entire outline of the fish 72. In a modified embodiment, as shown in Fig. 8, the outline of the fish as identified by the fish detector is segmented into sub-regions 88 that differ in distance from the camera 52. The distances in the different sub-regions 88 are calculated based on the ranging results obtained from the ranging detector 54. The distance values obtained by the individual detector elements of the ranging detector have already taken into account the effect of the angular deviation between the center line 70 of the field of view and the optical axis of the camera in the elevation direction. In addition, for each point within the outline of the fish 72, the effect of the angular deviation in the horizontal direction can be inferred from the position of the pixels on the fish in the field of view 76. Optionally, a further distance correction can be made for the relief of the fish body in the horizontal direction, which relief is at least generally known for the species of fish under consideration. Figure 9

[0106] Then, when a series of images is acquired from the fish 72 (e.g. at a frequency of 4 Hz as described above), the focus can be varied between the images so that the focus is adapted to the Figure 9 ​one of the sub-areas 88. This allows to obtain a high resolution image of all sub-areas 88 of the fish with a reduced depth of focus and thus with an aperture setting of the camera that requires a lower illumination light intensity.

[0107] Figure 10 A normalized image of the fish 72 is shown, which is finally submitted to the fish external parasite detector 86. This image can optionally contain several images of sub-areas 88 captured with different camera focal lengths. Furthermore, Figure 10 The image shown in Fig. 8 can be normalized in size to a standard size, which facilitates the comparison of captured images of fish with annotated images.

[0108] It will be observed that if the orientation of the fish is not at right angles to the optical axis of the camera, the image of the fish 72 as identified in Fig. 7 can be distorted (horizontally compressed). This results in a profile of the fish as shown in Fig. 8. Figure 9 The normalization process of the profile of the fish as shown in Fig. 8 can compensate for this distortion. Figure 10

[0109] Further, Figure 10 An optional embodiment is illustrated, in which the profile of the fish has been segmented into different areas 90, 92 and 94-100. The areas 90 and 92 allow the image processing system to distinguish between the top and bottom side of the fish, for which the color of the skin of the fish will be different on the one hand, and the intensity and spectrum of the illumination provided by the upper and lower illumination arrays 22 and 24 will be different on the other hand. Knowing the area 90 or 92 in which a pixel on the fish is located makes it easier for the fish external parasite detector 86 to search for characteristic features that form a contrast between the fish external parasites (e.g. sea lice) and the fish tissue.

[0110] Further areas 94-100 shown in this example designate selected anatomical features of the fish that are related to the characteristic population density of different species of fish external parasites (e.g. sea lice) on the fish. The same anatomical areas 94-100 will also be identified on the annotated images used for machine learning. This allows to train or configure the fish external parasite detector such that the confidence for detecting fish external parasites (e.g. sea lice) is adapted to the area that is currently being inspected.

[0111] Also, when the fish external parasite detector 86 is operated in an inference mode, it is possible to provide separate statistical information for the different areas 94-100 on the fish, which can provide useful information for identifying the species, gender and / or life stage of the external fish parasites (e.g. sea lice), and / or the extent of infestation.

[0112] Annotation, training, validation and testing

[0113] Figure 11 ​A flowchart showing the annotation of the detailed images and the training, validation, and testing of the detectors 84, 86 within the image processing system 78 is shown. The annotation and training process begins with the acquisition of the images. An annotation interface 102 allows a human to create a set of annotations that, when associated with the corresponding images, produce a corpus of annotated images.

[0114] In a preferred embodiment of the invention, the annotation interface communicates with a media server connected to the data store in the database 80. The annotation interface can be HTML-based, allowing a human annotator to load, view, and annotate images in a web browser. For each image, the annotator creates a polygon that encloses each visibly apparent fish, and a rectangular bounding box that encloses any ectoparasites (e.g., sea lice) present on the surface of the fish. Preferably, the annotation interface also allows the annotator to create a rectangular bounding box that encloses the fish eye (which can be visually similar to an ectoparasite, such as a sea lice). Preferably, the annotator also indicates the species, sex, and life stage of each louse.

[0115] The annotations created using the annotation interface 102 are stored in the database 80. When inserted and retrieved from the database, the annotations for a single image are serialized as a JSON object with a pointer to the relevant image. This simplifies ingestion of the annotated corpus by the machine learning program.

[0116] In a preferred embodiment of the invention, the machine learning program includes training a neural network on the corpus of annotated images. As Figure 11 shown, the annotated images can be divided into three groups of images. The first two groups of images are used to train and validate the neural network. More specifically, the first group of images (e.g., approximately 80% of the annotated images) is used to iteratively adjust the weights within the neural network. Periodically (i.e., after a certain number of additional iterations), the second group of images (approximately 10% of the annotated images) is used to validate the evolving detector to prevent overfitting. The result of the training and concurrent validation process is a trained detector 84, 86. The third group of images (e.g., approximately 10% of the annotated images) is used to test the trained detector. The testing procedure characterizes the performance of the trained detector, producing a set of performance metrics 104.

[0117] As Figure 11 shown, the entire training, validation, and testing process can be iterated multiple times as part of a more extensive neural network design process until acceptable performance metrics are obtained. As noted above, in a preferred embodiment of the invention, the process of Figure 11 is performed at least once to produce the fish detector 84, and at least once to produce the fish ectoparasite detector 86.

[0118] In alternative embodiments of the invention, to improve the quality of the training, validation and testing process, the machine learning program includes a data augmentation process to increase the size of the annotated corpus. For example, applying augmentation techniques such as noise addition and perspective transform to the human-annotated corpus can increase the size of the training corpus by up to 64 times.

[0119] Operation

[0120] Once the annotation, training, validation and testing process is complete, the detector can be run in inference mode to detect fish and fish ectoparasites, such as sea lice, in newly-acquired (unannotated) images. Figure 11

[0121] Specifically, each image to be processed is first passed to the fish detector 84. If the fish detector locates one or more patches within the image that it believes to be fish, the image is passed to the fish ectoparasite detector 86, such that the outline of the fish (and optionally, the regions 90-100) is delineated.

[0122] Figure 12 A flowchart is shown, which details the operation of the fish ectoparasite detector 86 within the image processing system 78 in inference mode. The image processing system first computes an image quality indicator for each acquired image. The quality indicator assesses the suitability of the image for detecting various life stages of fish ectoparasites (such as sea lice) on fish. In preferred embodiments of the invention, the image quality indicator comprises:

[0123] - the fraction of overexposed pixels. The fraction of pixels within the image whose luminance value exceeds the maximum allowable value (e.g. 250 for pixels with 8-bit depth)

[0124] - the fraction of underexposed pixels. The fraction of pixels within the image whose luminance value is below the minimum allowable value (e.g. 10 for pixels with 8-bit depth)

[0125] - a focus score; a measure of the quality of focus within the image, computed using the variance of the pixel values or the variance of the output of a Laplacian filter applied to the pixel values.

[0126] ​The acquired images, corresponding image quality indicators, and trained detectors from the training, validation, and testing procedures are passed to a detection operation that detects one or more classes of fish ectoparasites, e.g., sea lice (i.e., lice in a particular life stage), within the regions of the images that the fish detector identified (e.g., 90). The detections are then filtered based on the image quality indicators; if the image quality indicators indicate that the image quality is insufficient to allow reliable detection of a particular class of fish ectoparasite (e.g., sea lice), the detection of that class is excluded. When a detection is excluded, the image is excluded entirely from the calculation of the detection rate for that class. (i.e., the detection is excluded from the numerator of the detection rate, and the image is excluded from the denominator of the detection rate). The filtered detections can be stored in the detection database 106.

[0127] The image processing system 78 then combines the detections within the detection database with the performance indicators 104 from the training, validation, and testing procedures to model the statistical outcome of the population of fish ectoparasites, e.g., sea lice. Based on the known performance indicators, the machine vision system extrapolates from the detection rates in the acquired images to the actual incidence of fish ectoparasites, e.g., sea lice, in the fish population.

[0128] As noted above, the image processing system 78 can optionally use the fish location information determined by the ranging detector to inform its detection of fish. In particular, the fish detector can lower the confidence threshold required to detect a fish near the longitudinal centerline of each image report.

[0129] In embodiments of the application that incorporate multiple cameras with different orientations, the image processing system can also use the orientation information reported by the pose sensing unit 56 within the camera housing 20 at the time of image capture. For example, since fish typically swim parallel to the water surface, the orientation information can be used to bias the fish detection in favor of high-aspect-ratio patches with the long axis oriented perpendicular to the gravity vector. Determining the orientation of the fish relative to the camera also enables the image processing system with the fish detector to incorporate multiple neural networks, each trained for a particular fish orientation (e.g., for a relatively dark upper surface or a relatively light lower surface). The fish orientation can also inform the operation of the fish ectoparasite detector, since fish ectoparasites, e.g., sea lice, are more likely to be attached to particular locations on the fish.

[0130] End user interface

[0131] Finally, a system according to the application can include an end-user interface 108( Figure 1). The end user interface provides access to the results of the detection operations within the machine vision system. For example, a media server connected to the database 80 and data store can present images with machine-generated annotations indicating areas of the images where fish and fish external parasites (e.g., sea lice) were detected. The end user interface can also provide summary statistics (e.g., fish count, fish external parasite (e.g., sea lice) count, infestation rate) for a population of fish of interest (e.g., within an individual sea pen or across an entire farm).

[0132] In preferred embodiments of the invention, the end user interface also includes tools that make it easy to manage compliance. For example, the results of the detection operations of the image processing system can be automatically summarized and sent to regulatory agencies in a required form. The end user interface can also include predictive analytics tools, to predict the infestation rate of a population of fish, to predict the resulting economic impact, and to evaluate possible action plans (e.g., chemical treatment). The end user interface can also integrate with a wider set of aquaculture equipment (e.g., tools that monitor biomass, oxygen levels, and salinity levels).

[0133] Optionally, the end user interface includes the annotation interface 102 described above, allowing advanced end users to improve or extend the performance of the machine vision system. Also optionally, the end user interface can include an interface that allows parameters that regulate the behavior of the management camera and lighting control system to be adjusted.

Claims

1. A method for fish ectoparasite monitoring in aquaculture, comprising the steps of: - submerging a camera (52) into a sea pen (40) containing fish (72, 74), said camera having a field of view; - capturing images of the fish (72, 74) with the camera (52); and - identifying fish ectoparasites on the fish (72, 74) by analyzing the captured images, characterized in that: a target area within the field of view of the camera (52) is illuminated from above and below with light having different intensities and / or spectral composition, wherein an electronic image processing system (78) is used to detect fish (72, 74) in images captured by the camera (52), and to detect fish ectoparasites at a given location within the detected fish's contour; further comprising the steps of training a neural network on a corpus of annotated images to detect the fish ectoparasites; and using the trained neural network in the electronic image processing system (78).

2. The method according to claim 1, wherein the size of the target area illuminated by both the upper and lower lighting arrays is set to accommodate the entire contour of a fish.

3. The method according to claim 2, wherein the step of detecting the fish ectoparasites at a given location on a fish comprises distinguishing whether the given location is in a top side region (90) or a bottom side region (92) of the fish.

4. The method according to any one of claims 1-3, comprising the steps of: - operating a range finding detector (54) to continuously monitor a portion of the sea pen (40) to detect the presence of a fish in that portion of the sea pen, and to measure the distance from the camera (52) to the fish (72, 74) when a fish has been detected; and - calculating the focus setting of the camera (52) based on the measured distance when a fish has been detected; and - triggering the camera (52) when the detected fish (72, 74) is within a predetermined distance range.

5. The method according to claim 4, wherein the range finding detector (54) is used to measure the azimuth angle of the detected fish (72), and the measured azimuth angle is used by the electronic image processing system (78) to search for the contour of the fish in the captured images.

6. A system for fish ectoparasite monitoring in aquaculture, comprising: - a camera (52); - a lighting assembly (10) comprising an upper lighting array (22) and a lower lighting array (24); - a range finding detector (54); - an electronic image processing system (78) for detecting fish (72, 74) in images captured by the camera (52), and for detecting fish ectoparasites at a given location within the detected fish's contour; - wherein the electronic image processing system uses a trained neural network to detect the fish ectoparasites; and wherein the upper and lower illumination arrays each comprise one or more lights, wherein the lights in the upper illumination array have a different intensity and / or a different spectral composition than the lights in the lower illumination array.

7. The system according to claim 6, further comprising a posture sensing unit (56).

8. The system according to any of claims 6-7, wherein the ranging detector (54) continuously monitors a portion of the sea pen (40) to detect the presence of a fish in that portion of the sea pen and, when a fish has been detected, measures the distance from the camera (52) to the fish (72, 74); and - calculating a focus setting of the camera (52) based on the measured distance when a fish has been detected; and - triggering the camera (52) when the detected fish (72, 74) is within a predetermined distance range.

9. The system according to any one of claims 6-7, wherein the range finding detector (54) is configured to measure an azimuth angle of the detected fish (72), and the measured azimuth angle is used by the electronic image processing system (78) to search for the profile of the fish in the captured image.

Images