Method and system for fish ectoparasite monitoring in aquaculture

Abstract

A method for monitoring fish ectoparasites such as sea lice in aquaculture, comprising the steps of: - submerging a camera (52) into a sea pen (40) containing fish (72, 74); - capturing images of the fish (72, 74) with the camera (52); and - identifying fish ectoparasites such as sea lice on the fish (72, 74) by analyzing the captured images, characterized by the steps of: - distinguishing between at least two different categories of fish ectoparasites such as sea lice, which differ in the difficulty of identifying the fish ectoparasites such as sea lice; - calculating a quality indicator for each captured image, which allows identifying the category of fish ectoparasites such as sea lice for which the quality of the image is sufficient for sea lice detection; and - establishing a separate detection rate for each category of sea lice, each detection rate being based only on images for which the quality of the image, as described by the quality indicator, is sufficient for detecting the category of fish ectoparasites such as sea lice.

Description

[0001] This case is a divisional application of the applicant’s patent application No. 201880082815.5, filed on December 19, 2018, entitled “Method and System for Monitoring External Parasites in Fish in Aquaculture”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to a method for monitoring external parasites (such as sea lice) in fish during aquaculture, comprising the following steps:

[0003] - Submerge the camera in the sea pen containing the fish (72, 74);

[0004] - Capture images of the fish (72, 74) using the camera (52); and

[0005] - Identify fish ectoparasites such as sea lice on the fish (72, 74) by analyzing the captured images.

[0006] In this specification, the term "monitoring" refers to any activity aimed at providing an empirical basis for determining whether a given population of fish is infected with external parasites. The term monitoring may also include methods for determining the extent of infection of fish with external parasites. Although monitoring can be combined with measures to destroy or kill parasites, the term monitoring itself does not include such measures. Background Technology

[0007] Like humans and other animals, fish are susceptible to diseases and parasites. Parasites can be internal (endoparasites) or external (ectoparasites). Fish gills are a preferred habitat for many fish ectoparasites that attach to the gills but live outside them. The most common are monogenean copepods and certain groups of parasites, which can be extremely numerous. Other fish ectoparasites found on gills include leeches, and in marine environments, the larvae of isopods such as gnathiids. Most isopod fish parasites are external and feed on blood. Larvae and adults of the Gnathiidae family of cymothoids have piercing and sucking mouthparts, as well as clawed limbs adapted to attach tightly to their hosts. Cymothoa exigua is a parasite of a wide variety of marine fish. It causes the fish's tongue to atrophy and is believed to be the first instance of a parasite functionally replacing a host structure in an animal. Among the most common external parasites of fish is the so-called sea louse.

[0008] Sea lice are small parasitic crustaceans (family Sophiidae) that feed on the mucus, tissues, and blood of marine fish. Sea lice (plural sea louse) belong to the order Siphonostomatoida, family Sophiidae. There are approximately 559 species in 37 genera, including about 162 species in the genus *Lepeophtheirus* and 268 species in the genus *Caligus*. While sea lice are present in wild salmon populations, infestations in farmed salmon populations pose a particularly serious challenge. Several antiparasitic drugs have been developed for control purposes. The salmon lice (*L. salmonis*) is a major sea lice of concern in Norway. The Chilean lice (*Caligus rogercresseyi*) has become a major parasite of concern in Chilean salmon farms.

[0009] Sea lice have both free-swimming (plankton) and parasitic life stages. All stages are separated by molting. The developmental rate of salmon lice from egg to adult varies from 17 to 72 days depending on temperature. Eggs hatch into nauplius I, which molts into the second nauplius stage; neither nauplius stage involves feeding, relying on the energy reserves of the yolk and being adapted for swimming. The copepod larval stage is the infective stage, during which it searches for suitable hosts, possibly through chemical and mechanical sensory cues.

[0010] Once attached to a host, copepod larvae begin feeding and develop into the first nymph (chalimus) stage. Both copepods and nymphs possess a developed gastrointestinal tract and feed on the host's mucus and tissues within their attachment range. Pre-adult and adult sea lice (especially pregnant females) are aggressive feeders, and in some cases, feed on blood in addition to tissues and mucus.

[0011] The time and costs associated with mitigation efforts, along with fish mortality, increase fish production costs by approximately €0.2 per kilogram. Therefore, sea lice are a major concern for contemporary salmon farmers, who require significant resources to prevent infestations and comply with government regulations aimed at avoiding broader ecological impacts.

[0012] Effective mitigation (e.g., assessing the need and timing of vaccination or chemotherapy) and regulatory compliance both rely on the accurate quantification of fish ectoparasites (e.g., sea lice colonies in a single aquaculture operation). Currently, counting fish ectoparasites (e.g., sea lice) is entirely manual and therefore an extremely time-consuming process. For example, in Norway, counting and reporting must be done weekly, resulting in a direct cost of $24 million annually. Equally problematic is the questionable validity of statistics based on manual counting when extrapolating the counts of fish ectoparasites (e.g., adult female sea lice) on samples of 10 to 20 sedated fish to determine appropriate treatment for populations exceeding 50,000 fish. Consequently, both overtreatment and undertreatment are common.

[0013] WO2017 / 068127A1 describes a system of the type indicated in the preamble of claim 1, with the aim of automatically and accurately detecting and counting sea lice in a school of fish.

[0014] Any such system based on optical imaging must overcome several major challenges related to the marine environment and animal behavior.

[0015] - Optical distortion caused by density gradients. Turbulent mixing of warm and cold water, or especially salt and fresh water (e.g., in fjords), produces small-scale density changes, resulting in optical distortion. The effects are particularly severe on imaging objects smaller than 1–3 mm (e.g., sea louse larvae).

[0016] - Fish aversion to unfamiliar light sources. Fish may exhibit fear or more general aversion to light sources of unfamiliar location, intensity, or spectrum. Distortion in fish schools around such light sources typically increases the imager-to-fish distance, thus reducing the effective sensitivity of the imaging system. The cited literature addresses this problem by providing a guidance system for guiding fish along the desired imaging trajectory.

[0017] - Focus tracking in highly dynamic marine environments. Commercially available focus tracking systems perform poorly in highly dynamic scenes where a large number of fast-moving, seemingly realistic focus targets (i.e., a group of swimming fish) are simultaneously present in the field of view.

[0018] The purpose of this invention is to provide systems and methods for addressing these challenges and providing accurate, automated counting to reduce the amount of manual labor associated with fish ectoparasites (e.g., sea lice counting) and to enable more effective prediction and prevention of harmful infestations. Summary of the Invention

[0019] To achieve this objective, the method according to the present invention is characterized by the following steps:

[0020] - To distinguish between at least two different categories of fish ectoparasites, such as sea lice, based on the difficulty of identifying the fish ectoparasite, such as sea lice;

[0021] - Calculate a quality index for each captured image, the quality index allowing identification of fish ectoparasites such as sea lice of the aforementioned category, wherein the image quality is sufficient for the detection of fish ectoparasites such as sea lice; and

[0022] - Establish a separate detection rate for each category of fish ectoparasites such as sea lice, with each detection rate based solely on an image whose quality, as described by the quality metric, is sufficient for detecting that category of fish ectoparasites such as sea lice.

[0023] This invention helps avoid statistical artifacts that would otherwise occur due to the fact that certain categories of fish ectoparasites, such as sea lice (e.g., small larval fish ectoparasites like sea lice), escape detection simply because they are too small to be identified in images with poor image resolution. In this invention, statistics for easily detectable categories of fish ectoparasites, such as sea lice, can be based on large image samples, thus having low statistical noise, making it easier, for example, to detect changes in infestation levels over time. On the other hand, statistics for difficult-to-detect categories provide a more realistic picture of these categories, despite having slightly more statistical noise.

[0024] More specific optional features of the invention are indicated in the dependent claims.

[0025] Preferably, the system is capable of detecting and classifying ectoparasites such as sea lice in fish of both sexes at various sessile, active, and spawning life stages (e.g., larvae, pre-adults, adult males, adult oviparous females, and adult non-oviparous females).

[0026] Moreover, this system can form the basis of a comprehensive decision support platform for improving the operational performance, animal health, and sustainability of marine-based aquaculture. Attached Figure Description

[0027] The following are examples of embodiments, described in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 A side view of a camera and lighting rig according to a preferred embodiment of the present invention is shown;

[0029] Figure 2 This is a view of the sea fence, with floating structures within it. Figure 1 Accessories;

[0030] Figure 3 A front view of the camera and lighting accessories is shown;

[0031] Figure 4 A side view of the angular field of view of the rangefinder detector mounted on the accessory is shown;

[0032] Figure 5 and Figure 6 It is a chart illustrating the detection results of the rangefinder detector;

[0033] Figure 7-10 An image frame is shown, illustrating several steps in the image capture and analysis process;

[0034] Figure 11 A flowchart is shown, detailing the processes of training, validating, and testing a fish external parasite detector within an electronic image processing system (machine vision system) according to an embodiment of the present invention; and

[0035] Figure 12 A flowchart is shown, which details the operation of the fish external parasite detector in inference mode. Detailed Implementation

[0036] Image capture system

[0037] like Figure 1 As shown, the image capture system includes a camera and lighting accessory 10 and a camera and lighting control system 12, which enables the automatic acquisition of high-quality images of fish.

[0038] The camera and lighting accessories 10 include a vertical support member 14, an upper boom 16, a lower boom 18, a camera housing 20, an upper lighting array 22, and a lower lighting array 24. The camera housing 20 is connected to the vertical support member 14 and is preferably height-adjustable. The vertical positioning of the camera preferably ensures that the camera's field of view is at least partially (preferably mostly or entirely) covered by the illumination cones of the upper and lower lighting arrays 22, 24. Moreover, there is preferably a considerable angular offset between the center line of the camera's field of view and the center line of the illumination cones. This minimizes the amount of light backscattered (by particles in the water) to the camera, thereby maximizing the amount of light returning from the fish tissue. In the illustrated configuration, the camera can be mounted at a height between 1 / 4 and 3 / 4 of the length of the vertical support member, as measured from the lower end of the support member 14.

[0039] The upper boom 16 and lower boom 18 are connected to the vertical support member 14 at elbow joints 26 and 28, respectively, allowing the upper and lower booms to be angled and hinged relative to the vertical support member. The upper lighting array 22 and lower lighting array 24 are connected to the upper and lower booms at pivot joints 30 and 32, respectively, allowing the upper and lower lighting arrays to be angled and hinged relative to the upper and lower booms.

[0040] In the example shown, suspension rope 34 constitutes a double-line suspension for the camera and lighting accessory 10. The suspension rope allows control over the orientation of the accessory and can be connected to bracket 36 at different locations, thus maintaining the accessory's balance for a given configuration of booms 16 and 18. This allows for fine adjustment of the camera and lighting accessory's orientation (i.e., pitch angle) because the center of mass of the camera and lighting accessory is located below the connection point.

[0041] Preferably, the wiring conduit 38 carries all the data and power required for the upper lighting array, lower lighting array, and camera housing between the camera and lighting accessories and the camera and lighting control system 12.

[0042] Figure 2 A diagram illustrating a camera and lighting accessory 10 submerged in a seawall 40 is shown. The exemplary seawall shown is surrounded by a dock 42, from which vertical support members 44 extend upward. Tensioned cables 46 span between the support members. Suspension ropes 34 can be connected to the tensioned cables 46 to allow insertion and removal of the camera and lighting accessory 10 from the seawall, and to control the horizontal position of the accessory relative to the dock 42.

[0043] However, it should be noted that sea fences can also have different characteristics. Figure 2 The shape shown.

[0044] The extension of the support cables and ropes also allows for adjustment of the depth of the camera and lighting accessories below the water surface. Preferably, the camera and lighting accessories are positioned at a depth that positions the camera housing 20 below the surface mixing layer where the turbulent mixing of warm and cold water or salt and fresh water is most pronounced. This further reduces optical distortion associated with density gradients. The required depth varies based on location and season, but typically 2-3 m is preferred.

[0045] like Figure 3 As shown, the upper lighting array 22 and the lower lighting array 24 include a horizontal member 48 that supports one or more lighting units 50 within the lighting array along its length. Figure 3In the illustrated embodiment, the upper lighting array and the lower lighting array each include two lighting units 50; however, different numbers of lighting units can be used. The horizontal member 48 is connected to the upper and lower booms at pivotable joints 30, 32.

[0046] The elbow joints 26 and 28 between the vertical support member 14 and the upper and lower suspension rods 16 and 18, and the pivotable joints 30 and 32 between the upper and lower suspension rods and the horizontal member 48, collectively allow independent adjustment of the following:

[0047] - Horizontal offset between camera housing 20 and upper illumination array 22

[0048] - Horizontal offset between camera housing 20 and lower illumination array 24

[0049] - The angular orientation of the lighting units 50 within the upper lighting array 22, and

[0050] - The angular orientation of the lighting units 50 within the lower lighting array 24.

[0051] Typically, the upper and lower illumination arrays are positioned relative to the camera housing to provide sufficient illumination within the target area where the fish will be imaged for detection of external parasites such as sea lice. The longitudinally vertical design and configuration of the camera and illumination accessories 10 maximizes the likelihood that fish (which exhibit an aversion to long, horizontally oriented objects) will swim very close to the camera housing. Furthermore, the separate and independently adjustable upper and lower illumination arrays allow for the design of lighting schemes specifically tailored to address the unique lighting challenges of fish, as discussed in more detail below.

[0052] exist Figure 3 The camera housing 20, shown in the front view, includes a camera 52, a ranging detector 54 (e.g., a light-based time-of-flight detection and ranging unit), and an attitude sensing unit 56, which includes, for example, a magnetometer and an inertial measurement unit (IMU) or other known attitude sensing systems.

[0053] The camera is preferably a commercially available digital camera with a highly sensitive, low-noise sensor, capable of capturing clear images of fast-moving fish in relatively low light conditions. In a preferred embodiment of the invention, a Raytrix C42i camera is used, which provides a horizontal field of view of approximately 60° and a vertical field of view of approximately 45°. Of course, any other camera with similar performance (including electronically controllable focus) can be used as an alternative.

[0054] The rangefinder detector 54 is used to detect the range and orientation of fish swimming within the field of view of the camera 52. The detector includes a emitting optics 58 and a receiving optics 60. The emitting optics 58 generates a sector of light that is oriented vertically, but preferably collimated horizontally. That is, the sector diverges parallel to the vertical support member in pitch, but diverges with a relatively small yaw perpendicular to the vertical support member.

[0055] The receiving optics 60 includes an array of photodetector elements, each photodetector element detecting light incident from at least a portion of the acceptance angle spanning the vertical field of view of the camera. The angles of adjacent detector elements are pitch-adjacent to each other, collectively establishing an acceptance fan that completely covers the vertical field of view. This orientation and configuration of the transmitting and receiving optics is optimized to detect and locate fish swimming parallel to the horizontal water surface (which are typically high aspect ratio).

[0056] Preferably, the ranging detector 54 operates at the wavelength of light, thereby providing effective transmission in water. For example, blue or green light can be used to provide effective transmission in seawater. In a preferred embodiment of the invention, the ranging detector is Detectors, such as the LeddarTech M16, emit and receive 465nm light. Of course, the invention is not limited to this embodiment of the ranging detector.

[0057] Also in a preferred embodiment of the invention, the illumination sector generated by the emitting optics has a pitch divergence of approximately 45°, effectively spanning the camera's vertical field of view, and a yaw divergence of approximately 7.5°. The receiving optics 60 comprises an array of 16 detector elements, each having a field of view spanning approximately 3° of pitch and approximately 7.5° of yaw. Of course, the number of detector elements can be less than or greater than 16, but preferably not less than 4. Preferably, both the illumination sector and the receiving sector are horizontally centered within the camera's field of view, thereby ensuring that detected fish can be completely captured by the camera.

[0058] Systems with two or more rangefinders can also be envisioned. For example, a sector could be positioned “upstream” of the centerline (defined by the primary direction of fish movement) to provide an “advanced warning” of a fish entering the frame. Similarly, a unit could be placed downstream to confirm that a fish has left the frame.

[0059] The IMU in the attitude sensing unit 56 includes an accelerometer and a gyroscope, similar to those found in commercially available smartphones. In a preferred embodiment of the invention, the magnetometer and IMU are juxtaposed on a single printed circuit board within the camera housing 20. Together, the IMU and magnetometer measure the orientation of the camera housing (and thus the image acquired by the camera) relative to the water surface and seawall. Because fish typically swim parallel to the water surface and along the edge of the seawall, this information can be used to inform the machine vision system of the expected fish orientation in the captured image.

[0060] The upper illumination array 22 and the lower illumination array 24 may comprise one or more lamps of various types (e.g., incandescent lamps, gas discharge lamps, or LEDs) that emit light of any number of wavelengths. Preferably, a particular type of lamp is selected to provide sufficient color information (i.e., a sufficiently broad emission spectrum) to adequately contrast with fish tissue for external parasites (e.g., sea lice). Additionally, preferably, the type and intensity of the lamps within the upper and lower illumination arrays are selected to produce light of relatively uniform intensity reflected to the camera, despite the fish having a typically contrasting body with distinct light and dark areas.

[0061] In the embodiment presented here, the upper illumination array 22 includes a pair of xenon flash tubes. The lower illumination array 24 includes a pair of LEDs, each comprising a chip with 128 white LED dies. This hybrid illumination system provides a wider range of illumination intensities than could be achieved with a single type of illumination. Specifically, the flash tubes, synchronized with the camera shutter, provide brief but intense illumination (approximately 3400 lx) from above the fish. This ensures sufficient light reflected from the fish's typically dark, highly absorbent upper surface to the camera. (This requires a greater intensity of light than the LEDs in the lower illumination array can deliver.) Correspondingly, the LEDs provide sufficient illumination intensity for the fish's typically bright, highly reflective lower surface. (This requires an intensity lower than that provided by the xenon flash tubes in the upper illumination array.) The resulting uniform brightness reflected from the fish allows the camera to operate at lower sensitivity (e.g., below ISO 3200) to provide low-noise images to the machine vision system. Finally, both the xenon flash tubes and the LEDs provide a sufficiently broad spectrum to allow differentiation between fish tissue and external parasites such as sea lice.

[0062] As described above, the upper illumination array 22 and the lower illumination array 24 are positioned to provide the desired illumination throughout the target area. The target area is characterized by the camera's vertical field of view and near and far boundaries along the camera's axis. The distance from the camera to the near boundary is the greater of: (a) the camera's nearest accessible focal length, and (b) the distance a typical fish would travel across the camera's entire horizontal field of view. The distance from the camera to the far boundary is the distance at which the camera's angular resolution can no longer resolve the smallest fish external parasite (e.g., sea lice) that must be detected. Near boundary "a" and far boundary "b" are... Figure 4 The explanation is as follows.

[0063] Each lamp in the upper and lower illumination arrays provides a generally axisymmetric illumination pattern. Because there are multiple lamps in each array along the length of the horizontal member, the illumination pattern can be effectively characterized by the angular span in the pitch plane. The length of the vertical support member 14, the angular positions of the upper and lower booms 16 and 18, and the angular orientation of the upper and lower illumination arrays 22 and 24 are preferably adjusted such that the angular span of the upper and lower illumination arrays effectively covers the target area. The distance from the camera to the "sweet spot" depends on the size of the fish being monitored and, for example, can range from 200 mm to 2000 mm. In the case of salmon, for example, a suitable value could be approximately 700 mm.

[0064] In reality, the intensity of the illumination provided by the upper and lower illumination arrays is not perfectly uniform across their angular span. However, the method described above ensures an acceptable amount of illumination over the target area. It also produces an "optimal point" at a short distance near the boundary, where the angle of illumination between the upper and lower illumination arrays and the camera is optimal. This produces a best-lit image, thus also providing the best angular resolution achievable by the camera, with minimal impact from density gradient distortion.

[0065] A wide variety of other camera and lighting geometries can be used without departing from the scope of the invention. In particular, camera and lighting accessories can be configured and positioned in addition to Figure 1 Oriented in a direction other than the vertical. For example, cameras and lighting accessories can be oriented horizontally parallel to the water surface. Cameras and lighting accessories can also be configured to hold one or more cameras in a direction other than the vertical. Figure 1 A fixed position other than the indicated position (relative to the target area). Furthermore, some embodiments of the invention can integrate multiple cameras and lighting accessories, such as two cameras and lighting accessories symmetrically positioned in front of and behind the target area, thereby enabling the simultaneous capture of images of both sides of a single fish.

[0066] Camera and lighting control system

[0067] The camera and lighting control system 12 controls the operation of the image capture system. Camera and lighting control system:

[0068] - Receive and analyze data from the rangefinder 54 to determine the appropriate camera focal length.

[0069] - Control the camera's focus and shutter speed.

[0070] -Control the timing of the illumination of the upper illumination array 22 and the lower illumination array 24 relative to the shutter speed of the camera 52, and

[0071] - Receive, analyze, and store image data and image metadata, including measurement results from range detectors, magnetometers, and IMUs.

[0072] In this embodiment, the camera and lighting control system 12 includes a computer 62 and a power control unit 64, residing in a dry location (e.g., dock 42) physically adjacent to the camera and lighting accessory 10. In an alternative embodiment, at least a portion of the camera and lighting control system functionality provided by the computer is performed by an embedded system below the water surface (e.g., mounted to a vertical support member 14 or integrated within the camera housing). Typically, the computer 62 includes device drivers for each sensor within the camera and lighting accessory 10. Specifically, the computer includes device drivers for the magnetometer and IMU for the camera 52, the range detector 54, and the attitude sensing unit 56. The device drivers allow the computer to acquire measurement data from the relevant sensors and send control data to the relevant sensors. In a preferred embodiment, the measurement data and control data are exchanged between the means and processes running on the computer as messages within a Robot Operating System (ROS). Data from the sensors (including the range detector) arrives at a frequency of 10 Hz, while measurements from the magnetometer and IMU arrive at a frequency of 100 Hz. Each message is recorded on a disk on the computer.

[0073] Computer 62 provides control signals to power control unit 64 and optionally receives diagnostic data from power control unit. Power control unit provides power to upper lighting array 22 and lower lighting array 24 via cable 38. In a preferred embodiment, power control unit receives 220V AC power, which can be directly supplied to a charger for a capacitor bank to power the xenon flash tubes within upper lighting array 22 (when triggered). Power control unit transmits power to an underwater junction box (not shown), which converts AC power to DC power (e.g., 36V or 72V) for the LEDs within upper lighting array 24.

[0074] The focus calculation process, executed by computer 62, continuously monitors the ranging data to detect the presence of fish within the target area and determine their range. For each detector element, the ranging data consists of one or more distances from which light is reflected back to the detector element within its receiving angle in the receiving sector.

[0075] Figure 4 A side view of the angular field of view 66 of the detector element in the receiving optics 60 within the ranging receiving sector 68 is shown. As described above, the receiving angles of adjacent detectors are pitched adjacent to each other to establish the receiving sector. Figure 4 An array of 16 detectors is shown, each with a field of view spanning approximately 3° of pitch.

[0076] Figure 4 The average pitch angle of the detector within the ranging receiving sector 68 is shown. Each average pitch angle is illustrated by a center line 70 that bisects the field of view 66 of the corresponding detector. The average pitch angle is the angle between the bisecting center line 70 and the center line of the entire receiving sector 68, which is generally parallel to the optical axis of the camera 52.

[0077] Figure 4 A side view is also shown of the distance and average pitch angle of several detector elements blocked by fish 72, 74 within the ranging receiver sector 68. Typically, the focus calculation process detects a fish when several adjacent detector elements report similar distances. In a preferred embodiment of the invention, when M or more adjacent detector elements report similar distances d... i When a fish is detected, the number M can be in the range of 1 to 1 / 2 of the total number of detectors (i.e., 8 in this example). Specifically, the focus calculation process finds M or more neighboring distances d. i The adjacent set of [max(d)], for it, [d] i )-min(d i M and W are parameters that can be adjusted by the operator of the image capture system, where W represents the maximum permissible thickness, approximately corresponding to half the thickness of the largest fish to be detected. Depending on the size or age of the fish, parameters M and W are optimized for each system or fence. For each such detection, focus calculation will compute the average distance.

[0078] D=(1 / M)Σ1 M d i

[0079] and average azimuth

[0080] β=(1 / M)Σ1 M β i

[0081] Where β iThis is the average elevation angle of all adjacent detector elements. The focus calculation process then returns the focal length D. f =D*cosβ, which represents the distance from the camera along the camera's optical axis to the recommended focal plane.

[0082] Because of scattering particles in the water or objects that only block a portion of the detector's receiving angle (e.g., fish), multiple distances can be reported by a single detector. In practical implementations, in those cases where a single detector reports multiple distances, the farthest distance is used for focus calculation. This minimizes the number of false detections caused by particles in the water. In cases where multiple distances are actually associated with two fish (one of which only blocks a portion of the detector's receiving angle), it is likely that adjacent detectors will still successfully detect the partially blocked fish.

[0083] The presence and range information of the fish, determined by the focus calculation process, can be used to control the camera's image acquisition. For example, an image can only be captured if a fish is detected within a predetermined distance of the "optimal point" providing best lighting. For instance, if the "optimal point" is located at 700 mm, an image can only be captured if a fish is detected within a distance range of 600 to 800 mm. Whenever an image is captured, the camera and lighting control system sets the camera's focal length to the nearest range value determined by the focus calculation process.

[0084] In a preferred embodiment of the invention, the camera and lighting control system continuously and periodically triggers the camera to acquire images, for example at a frequency of 4 Hz or more typically at a frequency between 2 and 10 Hz. The focus calculation process continuously and periodically (e.g., at 10 Hz or more typically at 4 to 20 Hz) reports the current focal length and distance based on the latest fish detection, and the camera and lighting control system sets the camera's focal length to the latest available focal length.

[0085] When the camera shutter opens, the camera sends a synchronization signal to the camera and lighting control system 12, which is then transmitted to the power control unit 64. The power control unit illuminates the upper illumination array 22 and the lower illumination array 24 in sync with the shutter to ensure proper illumination of the captured image. In those embodiments of the invention where the lamps in the upper or lower illumination array cannot maintain a duty cycle equal to that of the camera (e.g., the xenon flash tube in a preferred embodiment of the invention), the power control unit may further include an illumination suppression process that continuously evaluates whether the power control unit should illuminate the upper and lower illumination arrays. In a preferred embodiment of the invention, illumination suppression occurs if (a) the emission history of the xenon flash tube in the upper illumination array is close to its thermal limit, or (b) the focus calculation process has not recently detected a flash and reports an updated range.

[0086] In a preferred embodiment of the invention, the lower-intensity LEDs in the lower illumination array illuminate for the duration of the camera exposure. The exposure length is set to the minimum length required for the LEDs to provide sufficient illumination. The flash length of the xenon flash tubes in the upper illumination array is adjusted to provide balanced illumination taking into account the contrasting brightness of typical fish.

[0087] The illumination provided by the LEDs in the lower illumination array is preferably sufficiently intense to provide a sufficiently short exposure to produce acceptable low motion blur within the captured image of the swimming fish. In a preferred embodiment of the invention, the sensors within the camera (particularly its pixel count), the camera's optics (particularly the angular span of the field of view), and the distance to the target area are selected to ensure that (a) the whole fish can be captured within the camera's field of view, and (b) even sufficient resolution of external parasites such as sea lice in juvenile fish. Providing 10 pixels per 2 mm (equivalent to the size of a juvenile sea lice) at a target distance spanning the width of a typical adult fish in the camera's 60° horizontal field of view requires 7.6 × 10 pixels per pixel. -3 The angular pixel spacing is °. For a fish swimming at a typical speed of 0.2 m / sec, the angular pixel spacing is less than 0.6 × 10⁻⁶. -3 A shutter speed of s ensures subpixel motion blur. To provide sufficiently low-noise images to the machine vision system, a sensor gain of less than ISO 3200 is preferred. This, in turn, requires illumination of approximately 3000 lux across the entire target area.

[0088] Figure 5 This explains that in Figure 4 The results were obtained using the range detector 54 in the depicted scenario. The results shown are from a detector element with a field of view of the centerline ranging from +6° to -15°. Figure 5 Each black dot in the diagram represents a detection event in which the reflected light has been received by the associated detector element. The position of the dot in the d-axis direction represents the distance to the detected object, calculated from the time it takes for the light signal to travel from the emitting optics 58 to the object and back to the receiving optics 60.

[0089] As previously stated, for multiple adjacent detectors, fish 72 and 74 are represented by detections at approximately the same distances d1 and d2, respectively. For each individual detector, the distance of the fish is the largest among the distances measured by that detector. Points at smaller distances represent noise caused by small particles in the receiving sector.

[0090] exist Figure 4 and Figure 5 As described above, fish 74 is partially obscured by fish 72, so that an image of the entire outline of the fish can only be obtained for fish 72 at a relatively small distance d1. Therefore, the camera's focal length will be adjusted to this distance d1.

[0091] Figure 6 This is a time graph showing the detection at distance d1 as a function of time t. It can be seen that the detection obtained from fish 72 (for an angle β ranging from -3° to -15°) is stable over a long time period corresponding to the time it takes for the fish to swim across receiving sector 68. Therefore, it is also possible to filter out noise by requiring the detection to be stable within a certain minimum time interval, or equivalently, by integrating the signal received from each detector element at a certain time and then thresholding the integration result.

[0092] In principle, having Figure 6 The type of detection history described herein may also be used to optimize the time intervals at which camera 52 captures a series of images, in order to ensure, on the one hand, that the number of images does not become unreasonably large, and on the other hand, that the series of images includes at least one image in which the entire fish is within the camera's field of view. For example, as Figure 6 As shown, a timer can be started at time t1 when a certain number of adjacent detector elements (three) detect an object that may be a fish. Then, the camera can be triggered at time t2 with a certain delay to start taking a series of images, and the series can be stopped at time t3 at the latest when the detector elements indicate that the tail of the fish has left the receiving sector.

[0093] Figure 7 This shows what happens when the fish 72's nose just crosses the receiving sector 68. Figure 6 The field of view of the camera at time t1 is 76.

[0094] Figure 8 This shows the camera 52 in a larger field of view than when the entire outline of the fish 72 is within the field of view 76. Figure 6 The image was captured at a later time, t2. At that moment, it could be obtained from... Figure 6 Based on the detection results of the detector element at β = -3° to -15°, it can be inferred that the centerline of the fish will be at β = -9°, such as... Figure 8 As shown. This information can be passed to the image processing system and can help identify the outline of the fish in the captured image.

[0095] Back Figure 1 The computer 62 of the camera and lighting control system 12 is connected to the image processing system 78, which accesses the database 80 through the data management system 82.

[0096] Data Management System

[0097] The automated system for detecting and counting fish ectoparasites (such as sea lice) also includes a data management system 82, which includes an interface that supports the acquisition, storage, searching, retrieval, and distribution of image data, image metadata, image annotations, and detection data created during the operation of the image processing system 78.

[0098] Data storage

[0099] The data management system 82 receives images from the image capture system, for example, in the form of ROS "packets". The data management system unpacks each packet into, for example, JPEG or PNG images and JSON (JavaScript Object Notation) metadata. The JPEG images are stored individually in the data storage area.

[0100] database

[0101] The JSON metadata unpacked from each ROS packet is stored in a database 80 associated with the data storage area. Typically, the metadata describes image capture parameters of the associated JPEG or PNG image. For example, the metadata includes an indication of the centroid pixel location within the outline of a fish detected by the LEDDAR unit (e.g., the pixel horizontally centered within the image, the longitudinal centerline of the fish). This pixel location can optionally be used by the image processing system (described in more detail below) to facilitate fish detection within the image.

[0102] Database 80 also stores annotation data created during the annotation process used to train image processing system 78 (described in more detail below). The database also stores information characterizing the location, size, and type of fish and fish ectoparasites (e.g., sea lice) detected by the machine vision system. Finally, the database stores authentication credentials that enable users to log in to various interfaces (e.g., the annotation interface or the end-user interface) via the authentication module.

[0103] Image processing system

[0104] In one embodiment, the present invention uses an image processing system 78 to perform the task of detecting fish ectoparasites (e.g., sea lice). In a preferred embodiment, separate neural networks are trained to provide a fish detector 84 and a fish ectoparasite detector 86. The fish detector 84 first detects individual fish within an image acquired by the image capture system. The fish ectoparasite detector 86 then detects individual fish ectoparasites, such as sea lice (if present), on the surface of each detected fish. Preferably, the fish ectoparasite detector also classifies the sex and life stage of each detected lice.

[0105] The detector is trained through a machine learning procedure that ingests a corpus of human-annotated images. The use of neural networks eliminates the need for explicitly defined characteristics of fish or fish ectoparasites (such as sea lice), such as breadth, shape, brightness, color, or texture, and instead directly utilizes knowledge of human annotators encoded within a corpus of the annotated images.

[0106] Figure 9 The position and outline of fish 72 in field of view 76 are shown, as detected by fish detector 84. Figure 4 , 7 The other fish 74 shown in Figure 8 has been excluded from consideration in this embodiment because it is partially obscured by fish 72. However, in the modified embodiment, it is also possible to detect fish 74 and look for external fish parasites (such as sea lice) on the skin of fish 74, as long as they are visible.

[0107] In one embodiment of the invention, the depth of focus of camera 52 has been selected to obtain a sharp image of the entire outline of fish 72. In a modified embodiment, such as Figure 9 As shown, the outline of the fish identified by the fish detector is segmented into sub-regions 88, which differ in distance from the camera 52. The distances in the different sub-regions 88 are calculated based on the ranging results obtained from the range detector 54. The distance values ​​obtained by the individual detector elements of the range detector already reflect the effect of the angular deviation in the pitch direction between the center line 70 of the field of view and the optical axis of the camera. Furthermore, for each point within the outline of the fish 72, the effect of the angular deviation in the horizontal direction can be inferred from the position of the pixels on the fish in the field of view 76. Optionally, additional distance corrections can be made for the horizontal relief of the fish body, which is at least generally known for the species of fish under consideration.

[0108] Then, when acquiring a series of images from fish 72 (e.g., at a frequency of 4Hz as described above), the focus can be varied between images, such that the focus is adapted to each image. Figure 9 One of the sub-regions 88 in the image. This allows for high-resolution images of all sub-regions 88 of the fish with reduced depth of focus, and therefore aperture settings for cameras requiring lower illumination intensity.

[0109] Figure 10 A standardized image of fish 72 is shown, which is ultimately submitted to the fish external parasite detector 86. This image may optionally include several images of sub-regions 88 captured with different camera focal lengths. Furthermore, Figure 10 The images shown can be standardized in size to a standard size, which facilitates comparison between images of fish captures and annotated images.

[0110] It will be observed that if the fish's orientation is not perpendicular to the camera's optical axis, then... Figure 9 Images of fish 72 identified in the image may be distorted (horizontal compression). This produces results such as... Figure 10 The standardization process of the fish outline shown can compensate for this distortion.

[0111] Furthermore, Figure 10 An optional implementation is described, in which the outline of the fish has been segmented into different regions 90, 92, and 94-100. Regions 90 and 92 allow the image processing system to distinguish between the top and bottom sides of the fish, for which, on the one hand, the skin color of the fish will be different, and on the other hand, the illumination intensity and spectrum provided by the upper and lower illumination arrays 22 and 24 will be different. Knowing which region 90 or 92 the pixels on the fish are located in makes it easier for the fish external parasite detector 86 to search for characteristic features that form a contrast between fish external parasites (e.g., sea lice) and fish tissue.

[0112] The additional regions 94-100 shown in this example specify selected anatomical features of the fish associated with characteristic population densities of different species of fish ectoparasites (such as sea lice) on the fish. The same anatomical regions 94-100 will also be identified on the annotated images used for machine learning. This allows the fish ectoparasite detector to be trained or configured such that the confidence level used to detect fish ectoparasites (such as sea lice) is suitable for the area currently being examined.

[0113] Moreover, when the fish external parasite detector 86 operates in inference mode, it is possible to provide separate statistical information for different regions 94-100 on the fish, which can provide useful information for identifying the species, sex and / or life stage, and / or degree of infection of external fish parasites (such as sea lice).

[0114] Annotation, training, validation, and testing

[0115] Figure 11 A flowchart detailing the annotation of images and the training, validation, and testing of detectors 84 and 86 within the image processing system 78 is shown. The annotation and training process begins with image acquisition. Annotation interface 102 allows humans to create a set of annotations, which, when associated with corresponding images, generate a corpus of annotated images.

[0116] In a preferred embodiment of the invention, the annotation interface communicates with a media server that stores data in database 80. The annotation interface may be HTML-based, allowing human annotators to load, view, and annotate images in a web browser. For each image, the annotator creates a polygon surrounding each clearly visible fish and a rectangular bounding box surrounding any external parasites (e.g., sea lice) present on the surface of the fish. Preferably, the annotation interface also allows the annotator to create a rectangular bounding box surrounding the fish's eye (which may visually resemble an external parasite, such as a sea lice). Preferably, the annotator also indicates the species, sex, and life stage of each louse.

[0117] Annotations created using annotation interface 102 are stored in database 80. During insertion and retrieval from the database, annotations for individual images are serialized into JSON objects via pointers to related images. This simplifies the ingestion of the annotated corpus by machine learning programs.

[0118] In a preferred embodiment of the invention, the machine learning procedure includes training a neural network on a corpus of annotated images. For example... Figure 11 As shown, the annotated images can be divided into three sets of images. The first two sets of images are used to train and validate the neural network. More specifically, the first set of images (e.g., approximately 80% of the annotated images) is used to iteratively adjust the weights within the neural network. Periodically (i.e., after a certain number of additional iterations), the second set of images (approximately 10% of the annotated images) is used to validate the evolving detector to prevent overfitting. The result of the training and simultaneous validation process is the trained detectors 84 and 86. The third set of images (e.g., approximately 10% of the annotated images) is used to test the trained detector. The testing procedure characterizes the performance of the trained detector, thereby generating a set of performance metrics 104.

[0119] like Figure 11 As shown, as part of a broader neural network design process, the entire training, validation, and testing process can be iterated multiple times until acceptable performance metrics are obtained. As described above, in a preferred embodiment of the invention, Figure 11 The process is executed at least once to generate fish detector 84, and at least once to generate fish external parasite detector 86.

[0120] In an alternative embodiment of the invention, to improve the quality of the training, validation, and testing processes, the machine learning program includes a data augmentation process to increase the size of the annotated corpus. For example, applying augmentation techniques such as noise addition and perspective transformation to a human-annotated corpus can increase the size of the training corpus by up to 64 times.

[0121] operate

[0122] Once completed Figure 11 The annotation, training, validation, and testing process allows the detector to run in inference mode to detect fish and fish ectoparasites, such as sea lice, in newly acquired (unannotated) images.

[0123] Specifically, each image to be processed is first passed to the fish detector 84. If the fish detector locates one or more patches that it considers to be fish within the image, the image is passed to the fish external parasite detector 86 so that the outline of the fish (and optionally, regions 90-100) is drawn.

[0124] Figure 12 A flowchart is shown, detailing the operation of a fish ectoparasite detector 86 in inference mode within an image processing system 78. The image processing system first calculates an image quality metric for each acquired image. The quality metric evaluates the suitability of the image for detecting various life stages of fish ectoparasites (e.g., sea lice) on fish. In a preferred embodiment of the invention, the image quality metric includes:

[0125] - Overexposed pixels. The portion of the image where the brightness value exceeds the maximum permissible value (e.g., 250 for a pixel with an 8-bit bit depth).

[0126] - Underexposed pixels. The portion of the image where the brightness value is below the minimum allowable value (e.g., 10 for a pixel with an 8-bit bit depth).

[0127] -Focus Score; a measure of focus quality within an image, calculated using the variance of pixel values ​​or the variance of the output of a Laplacian filter applied to pixel values.

[0128] Images acquired from training, validation, and testing procedures, along with corresponding image quality metrics and a trained detector, are passed to a detection operation that detects one or more categories of fish ectoparasites, such as sea lice (i.e., fish ectoparasites at a specific life stage, such as sea lice), within a region (e.g., 90°) of the image identified by the fish detector. Detections are then filtered based on image quality metrics; if the image quality metrics indicate that the image quality is insufficient to reliably detect a specific category of fish ectoparasites (e.g., sea lice), detection for that category is excluded. When excluding detections, the image is completely excluded from the detection rate calculation for that category (i.e., the detection is excluded from the numerator of the detection rate, and the image is excluded from the denominator of the detection rate). The filtered detections can be stored in a detection database 106.

[0129] The image processing system 78 then combines the detections in the detection database with performance metrics 104 from the training, validation, and testing procedures to model the statistical results of fish ectoparasite (e.g., sea lice) populations. Based on the known performance metrics, the machine vision system extrapolates the detection rate in the acquired images to the actual incidence of fish ectoparasites (e.g., sea lice) in the fish population.

[0130] As described above, the image processing system 78 may optionally use fish location information determined by the ranging detector to inform it of fish detection. Specifically, the fish detector may lower the confidence threshold required to detect fish near the longitudinal centerline of each image report.

[0131] In embodiments of the invention that integrate multiple cameras with different orientations, the image processing system can also use orientation information reported by the attitude sensing unit 56 within the camera housing 20 during image capture. For example, since fish typically swim parallel to the water surface, the orientation information can be used to bias fish detection towards high aspect ratio patches, where the long axis is oriented perpendicular to the gravity vector. Determining the fish's orientation relative to the camera also allows the image processing system with fish detectors to incorporate multiple neural networks, each trained for a specific fish orientation (e.g., for a relatively dark upper surface or a relatively light lower surface). Fish orientation can also inform the operation of the fish external parasite detector, as fish external parasites (e.g., sea lice) are more likely to attach to specific locations on the fish.

[0132] Final User Interface

[0133] Finally, the system according to the invention may include an end user interface 108. Figure 1 The end-user interface provides access to the results of detection operations within the machine vision system. For example, a media server connected to database 80 and the data storage can present images with machine-generated annotations indicating areas in the image where fish and fish ectoparasites (e.g., sea lice) have been detected. The end-user interface can also provide summary statistics (e.g., fish count, fish ectoparasite (e.g., sea lice) count, and infestation rate) for fish populations of interest (e.g., within an individual sea enclosure or across an entire fish farm).

[0134] In a preferred embodiment of the invention, the end-user interface also includes tools to facilitate management compliance. For example, the results of detection operations by the image processing system can be automatically summarized and sent to management authorities in the desired format. The end-user interface may also include predictive analytics tools to predict fish infestation rates, forecast the resulting economic impacts, and assess possible course of action (e.g., chemical treatment). The end-user interface can also be integrated with a wider set of aquaculture equipment (e.g., tools for monitoring biomass, oxygen levels, and salinity levels).

[0135] Optionally, the end user interface includes the aforementioned annotated interface 102, thereby allowing advanced end users to improve or extend the performance of the machine vision system. Also optionally, the end user interface may include an interface that allows adjustment of parameters managing the behavior of the camera and lighting control systems.

Claims

1. A method for fish ectoparasite monitoring in aquaculture, comprising the steps of: - submerging a camera (52) into a sea pen (40) containing fish (72, 74); - capturing images of the fish (72, 74) with the camera (52); and - identifying fish ectoparasites on the fish (72, 74) by analyzing the captured images, characterized by the steps of: - training a neural network to identify the contour of a fish in a captured image; - training a neural network to detect the fish ectoparasites within the contour of the fish; - using the trained neural networks to analyze the captured images; and - distinguishing between at least two different classes of fish ectoparasites; and a range detector (54) is used to measure the azimuth of a detected fish (72), and the measured azimuth is used in an electronic image processing system (78) to search for the contour of the fish in the captured images.

2. The method according to claim 1, comprising the steps of: - operating the range detector (54) to continuously monitor a portion of the sea pen (40) to detect the presence of a fish in that portion of the sea pen, and to measure the distance from the camera (52) to the fish (72, 74) when a fish has been detected; and - calculating the focus setting of the camera (52) based on the measured distance when a fish has been detected; and - triggering the camera (52) when a detected fish (72, 74) is within a predetermined distance range.

3. The method according to claim 2, wherein a target area defined by the field of view of the camera (52) and by the predetermined distance range is illuminated from above and below with light having different intensities and / or spectral composition.

4. The method according to claim 3, wherein the method further comprises the step of detecting the presence of fish ectoparasites in a specified area within the contour of the fish, which step comprises distinguishing whether the specified area is in a top side region or in a bottom side region of the fish.

5. A system for fish ectoparasite monitoring in aquaculture, the system comprising: - a camera (52) submerged into a sea pen (40) suitable for containing fish (72, 74), the camera being arranged to capture images of the fish; and - an electronic image processing system (78) configured to identify fish ectoparasites on the fish by analyzing the captured images, wherein the system is configured to perform the method according to any one of claims 1 to 4.

6. The system according to claim 5, wherein The electronic image processing system (78) comprises: a fish detector (84) configured to identify the contour of a fish in a captured image; and a fish ectoparasite detector (86) configured to detect fish ectoparasites in a specified area within the contour of the fish (72).

7. The system according to claim 6, wherein the electronic image processing system (78) comprises a neural network trained to identify the fish ectoparasites.

8. The system according to claim 6 or 7, wherein the electronic image processing system (78) comprises a neural network trained to provide the fish detector (84).

9. The system according to any one of claims 5 to 7, comprising: - a range detection detector (54) configured to detect the presence of a fish and to measure the distance from the range detection detector to the fish, which is mounted adjacent to the camera (52); and - a camera and illumination control system (12) arranged to control the focal length of the camera (52) based on the measured distance and to trigger the camera (52) when a fish is detected within a predetermined distance range.

10. The system according to claim 9, wherein the range detection detector (54) comprises a light time-of-flight based range and detection unit having: - LED emission optics (58) for emitting light; and - receiving optics (60) adapted to receive reflected light and to measure the time of flight of the received light.

11. The system according to claim 10, wherein the LED emission optics (58) are configured to emit light in the form of a sector that spreads over an extended angular range in the vertical direction and is collimated in the horizontal direction.

12. The system according to claim 11, wherein the receiving optics (60) comprise a plurality of detector elements having adjacent fields of view (66) that together form a vertically oriented acceptance sector (68), the range detection detector being adapted to measure an azimuth angle (β) under which the reflected light is detected by an individual detector element.

13. The system according to any one of claims 5 to 7, comprising a camera and illumination assembly (10) having a vertical support member (14), an upper boom (16) hinged to an upper end of the vertical support member (14) and carrying an upper illumination array (22), a lower boom (18) hinged to a lower end of the vertical support member (14) and carrying a lower illumination array (24), and a housing (20) connected to the vertical support member and carrying the camera (52), wherein the upper and lower illumination arrays (22, 24) are configured to illuminate a target area within the field of view of the camera (52) from above and from below.

14. The system according to claim 13, wherein the upper illumination array (22) is configured to emit light of a different intensity and / or spectral composition than the light emitted by the lower illumination array (24).

15. The system according to claim 14, wherein the upper illumination array (22) comprises a flash illumination unit and the lower illumination array (24) comprises an LED illumination unit.

16. The system according to claim 13, comprising a pose sensing unit (56) adapted to detect the pose of the camera and illumination assembly (10) relative to the sea pen (40). ​ 17. The system of claim 9, wherein the camera and lighting control system (12) is configured to control the camera (52) to capture a series of images over a long time interval, the ranging detector (54) continuously detecting the fish within the predetermined distance range over the long time interval.

Images