System for monitoring external parasites of fish in aquaculture

Abstract

FIELD: aquaculture.SUBSTANCE: group of inventions can be used to monitor external parasites of fish. The filming apparatus (52) is immersed in a sea farm (40) comprising fish (72, 74), and the filming apparatus has a field of view. Images of the fish (72, 74) are captured using a camera (52); and an external fish parasite on the fish (72, 74) is identified by analysing the captured images. The target area is illuminated in the field of view of the camera (52) from above and below. To detect fish (72, 74) in the image captured by the camera (52), an electronic image processing system (78) is used. A neural network is trained to identify an external parasite of fish and the trained neural network is used in an electronic image processing system (78). The range-determining detector (54) is controlled to continuously monitor a portion of the sea farm (40) to detect the presence of fish in that portion of the sea farm and, when a fish is detected, the distance from the filming apparatus (52) to the fish (72, 74) is measured. When the fish is detected, the focusing settings of the camera (52) are calculated based on the measured distance; and the camera (52) is launched when the detected fish (72, 74) is within a predetermined distance range. The range detector (54) is used to measure the bearing angle of the detected fish (72), and the measured bearing angle is used in the image processing system (78) to search for the silhouette of the fish in the captured image. A system for monitoring external parasites of fish is also proposed.EFFECT: automated counting of fish parasites, predicting and preventing fish infestations with parasites.9 cl, 12 dwg

Description

[0001] Field of technology

[0002] The invention relates to a system for monitoring external parasites of fish such as sea lice in aquaculture, the system comprising:

[0003] - a filming apparatus suitable for diving into a marine farm suitable for keeping fish, wherein the filming apparatus is adapted to capture images of fish; and

[0004] - An electronic image processing system configured to identify external fish parasites such as sea lice on fish by analyzing captured images.

[0005] In this specification, the term "monitoring" refers to any activity aimed at developing an empirical basis for deciding whether a given fish population is infected with external parasites. The term "monitoring" can also encompass a method for determining the extent to which fish are infected with external parasites. Although monitoring can be combined with measures to eliminate or kill parasites, the term "monitoring" alone does not encompass such measures.

[0006] State of the art

[0007] Like humans and other mammals, fish suffer from diseases and parasites. Parasites can be internal (endoparasites) or external (ectoparasites). Fish gills are the preferred habitat for many external fish parasites, which attach to the gills but live outside them. The most common are monogenetic flukes and certain groups of parasitic copepods, which can be exceptionally numerous. Other external fish parasites found on the gills include leeches and, in seawater, gnathiid (isopod) larvae. Isopod parasites of fish are most often external and feed on blood. Leeches of the family Gnathiidae and sexually mature cymothodids have piercing and sucking mouthparts and clawed limbs adapted for clinging to their hosts. Cymothoa exigua is a parasite of various marine fish.It causes atrophy of the fish's tongue and takes its place, thus being considered the first case of a parasite that replaces its host structure in animals. Among the most common external parasites of fish are the so-called sea lice.

[0008] Sea lice are small parasitic crustaceans (family Caligidae) that feed on the mucus, tissue, and blood of marine fish. Sea lice (plural sea lice) are members of the order Siphonostomatoida of the family Caligidae. There are approximately 559 species in 37 genera, including approximately 162 species of Lepeophtheirus and 268 species of Caligus. Although sea lice are present in numerous wild salmon populations, sea lice infestations are particularly problematic in farmed salmon populations. Several antiparasitic drugs have been developed to combat this scourge. The main sea louse species of concern in Norway is the salmon louse (L. salmonis). The main parasite of concern on salmon farms in Chile is the nauplii (Caligus rogercresseyi).

[0009] Sea lice have both a free-swimming (planktonic) stage and a parasitic stage. These stages are separated by molts. The development rate of L. salmonis from egg to adult varies from 17 to 72 days, depending on temperature. The eggs hatch into nauplius I larvae, which molt to enter the second naupliar stage. Both naupliar stages rely on yolk reserves for energy and are adapted to swimming. The copepod stage is the infective stage, searching for a suitable host, likely based on chemosensory and mechanosensory cues.

[0010] Immediately after attaching to the host, the copepod stage begins feeding and develops into the first chalimus stage. The copepod and chalimus stages have a developed gastrointestinal tract and feed on mucus and tissue within the area where they attach. Immature and mature sea lice, especially egg-bearing females, are aggressive phages, sometimes feeding on blood in addition to tissue and mucus.

[0011] The time and costs associated with cleanup efforts and increased fish mortality increase the cost of fish production by approximately 0.2 euros / kg. Consequently, external fish parasites such as sea lice are a major concern for modern salmon farmers, who devote significant resources to preventing infestations and complying with government regulations aimed at preventing wider environmental impacts.

[0012] Both effective mitigation (e.g., assessing the need and timing of vaccination or chemical cleaning) and regulatory compliance depend on the accurate quantification of external fish parasites, such as sea lice populations, within individual fish farming operations. Currently, counting external fish parasites, such as sea lice, is a completely manual process and therefore time-consuming. For example, in Norway, counts must be made and reported weekly, which alone results in an annual direct cost of US$24 million. Equally troublesome is the questionable reliability of statistics based on manual counts, when counting external fish parasites, such as mature female sea lice, based on sample data from 10 to 20 anesthetized fish is extrapolated to determine appropriate treatment for fish populations over 50,000 fish.As a result, both over- and under-processing are common.

[0013] Document WO 2017 / 068127 A1 describes a system of the type referred to in the preamble to claim 1, aimed at creating the ability to automatically and accurately detect and count external fish parasites such as sea lice within fish populations.

[0014] Any such system based on optical imaging must overcome several significant challenges associated with the marine environment and animal behavior.

[0015] Optical distortion due to density gradients. Turbulent mixing of warm and cold water, or especially salt and fresh water (e.g., within fjords), generates small-scale density variations that cause optical distortion. This is particularly severe when imaging objects smaller than 1-3 mm (e.g., juvenile sea lice).

[0016] Fish aversion to unfamiliar light sources. Fish may exhibit a fear or more general aversion response to unfamiliar light sources in unfamiliar locations, with unfamiliar intensities, or with unfamiliar spectra. Distortion of fish schools around such a light source will generally increase the typical distance from the imaging device to the fish, reducing the effective visual acuity of the imaging system. The cited paper proposes a solution to this problem by developing a guidance system to direct the fish along the desired imaging trajectory.

[0017] Focus tracking in highly dynamic marine environments. Commercially available focus tracking systems do not perform adequately in rapidly changing environments where a large number of fast-moving, adequately focused targets (e.g., a school of swimming fish) are simultaneously present within the field of view.

[0018] The object of the present invention is to develop a system and method aimed at solving these problems and providing accurate automated counts to reduce the amount of human labor associated with external fish parasite such as sea lice counts, and to enable more effective prediction and prevention of harmful parasite infestations.

[0019] The essence of the invention

[0020] To solve this problem, the system according to the invention is characterized in that it contains:

[0021] a range-determining detector configured to detect the presence of fish and measure the distance from said detector to the fish, and which is installed near the surveying apparatus); and

[0022] an electronic control system configured to control the focus of the filming apparatus based on the measured distance and to trigger the filming apparatus when a fish is detected within a certain predetermined distance range.

[0023] Instead of attempting to guide the fish along a specific trajectory, the inventive approach deploys the surveying apparatus only when the fish is detected within a certain range. This allows the fish to behave naturally within the school, and the surveying apparatus and range-determining detector can be positioned near a path the fish are likely to follow due to their natural behavior, such as along the boundary of a marine farm.

[0024] Because the system only captures images when a fish is actually detected, the number of images to be captured and, consequently, the number of images to be analyzed can be limited. Furthermore, since lighting is only required when the image is actually captured, fish irritation from light sources is also reduced.

[0025] The ability of the distance detector to provide reliable distance data is used to more accurately control the focus of the camera, thereby improving the quality of the captured images.

[0026] More specific features of the invention, optionally provided, are indicated in dependent claims of the invention formula.

[0027] In a preferred embodiment, the system is capable of detecting and categorizing sea lice of both sexes at various life stages, including immobile, mobile, and spawning (e.g., juveniles, immatures, mature males, spawning mature females, and non-spawning mature females).

[0028] In addition, the system could form the basis for an embedded decision support platform that enhances operational performance, improves animal health, and ensures sustainable development of ocean-based aquaculture.

[0029] Brief description of the drawings

[0030] Examples of embodiments of the invention will now be described with reference to the drawings, in which:

[0031] Fig. 1 is a side view of the assembly of the filming apparatus and lighting equipment in accordance with the preferred embodiment of the invention;

[0032] Fig. 2 - view of a marine farm with equipment suspended on it in accordance with Fig. 1;

[0033] Fig. 3 - front view of the filming apparatus and lighting equipment unit;

[0034] Fig. 4 - side view of the angular field of view of the range-determining detector installed on the equipment;

[0035] Fig. 5 and 6 are diagrams illustrating the results of detection by means of a range-determining detector;

[0036] Fig. 7-10 - image frames illustrating several stages of the image capturing and analysis procedure;

[0037] Fig. 11 is a flow chart detailing the process of annotating images and training, validating and testing a fish external parasite detector within an electronic image processing system (machine vision system) in accordance with an embodiment of the invention; and

[0038] Fig. 12 - A flow chart detailing the operation of the fish external parasite detector in inference mode.

[0039] Detailed description of the invention

[0040] Image capture system

[0041] As shown in Fig. 1, the image capturing system includes a camera and lighting equipment unit 10 and a camera and lighting control system 12, which can automatically obtain high-quality images of fish.

[0042] The assembly 10 of the filming apparatus and lighting equipment comprises a vertical support element 14, an upper boom 16, a lower boom 18, a housing 20 of the filming apparatus, an upper lighting matrix 22 and a lower lighting matrix 24. The housing 20 of the filming apparatus is attached to the vertical support element 14 and is preferably adjustable in height. The vertical positioning of the filming apparatus is preferably such that the field of view of the filming apparatus is at least partially (preferably mostly or completely) covered by the light cones of the upper and lower lighting matrices 22, 24. In the preferred embodiment, there is also a significant angular offset between the geometric axis of the field of view of the filming apparatus and the geometric axes of the light cones. This minimizes the amount of light scattered (by particles in the water) back into the camera, while maximizing (relatively) the amount of light returning from the fish tissue.With the shown setup, the filming apparatus can be installed at a certain height, measured from the end of the support element 14 and located in the range between 1 / 4 and 3 / 4 of the length of the vertical support element.

[0043] The upper boom 16 and the lower boom 18 are articulated with the vertical supporting member 14 at elbow joints 26 and 28, respectively, which provide angular articulated joints of the upper boom and the lower boom relative to the vertical support member. The upper lighting matrix 22 and the lower lighting matrix 24 are articulated with the upper boom and the lower boom at rotary joints 30 and 32, respectively, which provide angular articulated joints of the upper lighting matrix and the lower lighting matrix relative to the upper boom and the lower boom.

[0044] In the example shown, suspension cables 34 form a bifilar suspension for the assembly 10 of the camera and lighting equipment. The suspension cables allow for azimuth control of the equipment and can be attached to bracket 36 in various positions, thereby maintaining the equipment's balance within a given configuration of booms 16 and 18. This enables precise adjustment of the orientation (i.e., the installation angle) of the camera and lighting equipment assembly, since the center of mass of the camera and lighting equipment assembly is below the attachment point.

[0045] In the preferred embodiment, the transfer of all data and power required by the upper lighting array and the lower lighting array occurs through a cable hose 38, which also carries the camera housing and extends between the camera and lighting assembly and the camera and lighting control system 12.

[0046] Figure 2 shows a drawing of the unit 10 of the filming apparatus and lighting equipment, loaded into the offshore farm 40. The shown possible offshore farm is surrounded by a dock 42, from which vertical support elements 44 extend upward. Between the support elements, tensioned cables 46 pass. Suspension ropes 34 can be attached to the tensioned cables 46, providing for the introduction of the unit 10 of the filming apparatus and lighting equipment into the offshore farm and their removal from it, as well as control of the horizontal position of the equipment relative to the dock 42.

[0047] At the same time, it should be noted that the marine farm may also have a shape different from that shown in Fig. 2.

[0048] The length of the supporting cables and ropes also allows for adjustment of the depth of the camera and lighting assembly below the water surface. Preferably, the camera and lighting assembly is positioned at a depth such that the camera housing 20 is below the surface mixing layers, where the turbulent mixing of warm and cold water or salt and fresh water is most pronounced. This further reduces optical distortion associated with density gradients. The required depth depends on location and time of year, but typically a depth of 2–3 m is preferred.

[0049] As shown in Fig. 3, the upper lighting matrix 22 and the lower lighting matrix 24 comprise horizontal members 48 that support one or more lighting fixtures 50 within the lighting matrix along their length. In the embodiment shown in Fig. 3, each of the upper lighting matrix and the lower lighting matrix comprises two lighting fixtures 50; however, different numbers of lighting fixtures can be used. The horizontal members 48 are articulated with the upper boom and the lower boom at pivot joints 30, 32.

[0050] The elbow joints 26, 28 between the vertical support member 14 and the upper boom 16 and the lower boom 18 and the rotary joints 30, 32 between the upper boom and the lower boom and the horizontal members 48 jointly provide independent adjustment:

[0051] - horizontal displacement between the casing 20 of the filming apparatus and the upper lighting matrix 22;

[0052] - horizontal displacement between the casing 20 of the filming apparatus and the lower illumination matrix 24,

[0053] - angular orientation of the lighting devices 50 within the upper lighting matrix 22; and

[0054] - angular orientation of lighting devices 50 within the lower lighting matrix 24.

[0055] In general, the upper illumination array and the lower illumination array are positioned relative to the camera housing so as to provide adequate illumination within a certain target area where the fish will be imaged to detect external fish parasites such as sea lice. The vertical longitudinal design and configuration of the camera assembly 10 and the lighting equipment maximizes the likelihood that fish (which exhibit an aversion to long, horizontally oriented objects) will swim in close proximity to the camera housing. In addition, separate and independently adjustable upper illumination arrays and lower illumination arrays provide lighting algorithms designed specifically to address the lighting issues specific to fish, discussed in more detail below.

[0056] The camera housing 20, which is shown by means of a front view in Fig. 3, comprises a camera 52, a ranging detector 54, for example, a ranging unit using light and detecting its propagation time, and a position sensor 56, including, for example, a magnetometer and an inertial measurement unit (IMU), or other known position determination systems.

[0057] The camera is preferably a commercially available digital camera with a high-sensitivity, low-noise sensor, capable of capturing clear images of fast-swimming fish in relatively low light. In one preferred embodiment, the Raytrix C42i camera is used, providing a horizontal field of view of approximately 60° and a vertical field of view of approximately 45°. Of course, any other camera with similar features (including electronic focus control) can be used as an alternative.

[0058] Range-determining detector 54 is used to detect the range and bearing of fish swimming within the field of view of survey apparatus 52. This detector comprises emitting optical means 58 and receiving optical means 60. The emitting optical means 58 creates a light fan oriented in the vertical direction, but preferably collimated in the horizontal direction. That is, the fan diverges in the pitch direction - parallel to the vertical support element, but diverges relatively little in the yaw direction - perpendicular to the vertical support element.

[0059] The receiving optical means 60 comprises an array of light detector elements, each of which detects light incident within a reception angle whose resolution occupies at least a portion of the vertical field of view of the imaging apparatus. The corners of adjacent detector elements are adjacent to each other along the pitch, together forming a reception fan that completely covers the vertical field of view. This orientation and configuration of the transmitting and receiving optical means is optimized for detecting and locating fish bodies (which typically have a high length-to-width ratio) swimming parallel to the horizontal surface of the water.

[0060] In a preferred embodiment, the ranging detector 54 operates at a wavelength of light that ensures its effective transmission within water. For example, blue light or green light can be used to ensure effective transmission within seawater. In a preferred embodiment, the ranging detector is a LEDDAR detector. ® (Light-Emitting-Diode Detection and Ranging detector), such as the LeddarTech M16, which emits and receives light at a wavelength of 465 nm. Of course, the invention is not limited to this embodiment of a ranging detector.

[0061] In another preferred embodiment of the invention, the illumination fan generated by the emitting optical means diverges by approximately 45° in pitch, which essentially provides an aperture that covers the vertical field of view of the filming apparatus, and diverges by approximately 7.5° in yaw. The receiving optical means 60 comprises an array of 16 detector elements, each with a field of view whose aperture occupies approximately 3° in pitch and approximately 7.5° in yaw. Of course, the number of detector elements may be less or more than 16, and is preferably at least 4. In the preferred embodiment, both the illumination fan and the detection fan are horizontally centered within the field of view of the filming apparatus, ensuring that the filming apparatus can completely capture the detected fish.

[0062] Systems with two or more range-determining detectors could also be considered. For example, a fan could be positioned "upstream" (as determined by the fish's predominant swimming direction) from the geometric axis to provide "early warning" of a fish entering the frame. Similarly, a unit could be positioned "downstream" to confirm the fish's exit from the frame.

[0063] The IMU in position sensor 56 contains an accelerometer and gyroscope, similar to those found in commercially available smartphones. In one preferred embodiment, the magnetometer and IMU are located together on a single printed circuit board within the housing 20 of the camera. The IMU and magnetometer jointly measure the orientation of the camera housing (and therefore the images acquired by the camera) relative to the water surface and the marine farm. Since fish generally swim parallel to the water surface and along the edges of the marine farm, this information can be used to inform the machine vision system of the expected orientation of the fish within the acquired images.

[0064] The upper illumination array 22 and the lower illumination array 24 may include one or more lamps of various types (e.g., incandescent, gas discharge, or LED) emitting light at any number of wavelengths. Preferably, specific lamp types are selected to provide sufficient color information (i.e., a sufficiently broad emission spectrum) to adequately distinguish external fish parasites, such as sea lice, from fish tissue. Furthermore, the types and intensities of the lamps within the upper illumination array and the lower illumination array are preferably selected to provide a relatively uniform light intensity reflected to the camera, despite the typical, noticeable back-shadow effect of the fish body.

[0065] In the embodiment proposed here, the upper illumination matrix 22 contains a pair of xenon flash lamps. The lower illumination matrix 24 contains a pair of LED lamps, each of which contains a chip with 128 crystals of white light-emitting diodes (LEDs). This hybrid illumination system provides a greater range of illumination intensities than that which can be achieved using a single type of illumination. In particular, the flash lamps provide short but intense illumination (approximately 3400 lux) of the fish from above, synchronized with the operation of the shutter of the camera. This ensures adequate light reflected to the camera from the generally dark, highly light-absorbing upper surfaces of the fish. (This requires a greater light intensity than that which could be provided by the LED lamps of the lower illumination matrix.) Accordingly, LED lamps provide adequate illumination intensity for the typically light-colored undersurfaces of fish. (This requires an intensity lower than that provided by xenon flash lamps in the upper illumination array.) The resulting uniformly bright light reflected from the fish allows the camera to operate at relatively low sensitivities (e.g., below ISO 3200), providing low-noise images for the machine vision system. Finally, both xenon flash lamps and LED lamps provide an adequately broad spectrum, allowing the differentiation of external fish parasites such as sea lice from fish tissue.

[0066] As described above, the upper illumination matrix 22 and the lower illumination matrix 24 are arranged to provide the desired illumination over the entire target area. The target area is characterized by the vertical field of view of the camera, as well as the near and far boundaries along the axis of the camera. The distance from the camera to the near boundary is the greater of (a) the closest achievable focal length of the camera and (b) the distance at which a typical fish blocks the entire horizontal viewing angle of the camera. The distance from the camera to the far boundary is the distance at which the angular resolution of the camera no longer allows the detection of the smallest external parasite of fish, such as sea lice, which should be detected. The near boundary "a" and the far boundary "b" are illustrated in Fig. 4.

[0067] Each of the lamps within the upper illumination matrix and the lower illumination matrix generally provides an axisymmetric illumination pattern. Since multiple lamps are located within each matrix along the length of the horizontal elements, the illumination pattern can be effectively characterized by the angular aperture in the trim plane. The length of the vertical support element 14, the angular position of the upper boom 16 and the lower boom 18, and the angular orientation of the upper illumination matrix 22 and the lower illumination matrix 24 are preferably adjusted so that the angular aperture of the upper illumination matrix and the lower illumination matrix effectively covers the target area. The distance from the camera to the "sweet spot" depends on the size of the fish to be monitored and can range, for example, from 200 mm to 2000 mm. For salmon, for example, a suitable value may be approximately 700 mm.

[0068] In practice, the illumination intensities provided by the upper and lower illumination arrays are not completely uniform across their entire angular aperture. However, the above approach ensures that an acceptable illumination level is provided across the target area. This also results in a "sweet spot" located a short distance beyond the near boundary, where the illumination angle between the upper and lower illumination arrays and the camera is optimal. This results in optimally illuminated images, providing the best angular resolution achievable by the camera and minimally suffering from distortion due to density gradients.

[0069] Within the scope of the invention, a wide range of other camera systems and a wide variety of lighting geometries can be used. In particular, the camera system and lighting assembly can be designed to be positioned in orientations other than the vertical orientation shown in Fig. 1. For example, the camera system and lighting assembly can be oriented horizontally—parallel to the water surface. The camera system and lighting assembly can also be designed to support one or more camera systems in fixed positions (relative to the target area) other than those shown in Fig. 1.Additionally, some embodiments of the invention may provide for multiple camera and lighting units, such as two camera and lighting units symmetrically positioned in front of and behind the target area, allowing for simultaneous image capture from both sides of a single fish.

[0070] Camera and Lighting Control System

[0071] The camera and lighting control system 12 controls the operation of the image capture system. Camera and lighting control system:

[0072] - Receives and analyzes data from the range detector 54, determining the proper focal length of the camera;

[0073] - controls the focus and shutter of the camera;

[0074] - controls the synchronization of the illumination of the upper illumination matrix 22 and the lower illumination matrix 24 relative to the shutter of the camera 52; and

[0075] - Receives, analyzes and stores image data and image metadata, including measurements obtained from the ranging detector, magnetometer and IMU.

[0076] In this embodiment, the camera and lighting control system 12 comprises a computer 62 and a power control unit 64, which are located in a dry location (e.g., dock 42) physically close to the camera and lighting equipment assembly 10. In alternative embodiments, at least part of the functionality of the camera and lighting control system provided by the computer is implemented by a system submerged under the surface of the water (e.g., mounted on a vertical support element 14 or built into the housing of the camera). In general, the computer 62 includes device drivers for each sensor within the camera and lighting equipment assembly 10. In particular, the computer includes device drivers for the camera 52, the range-determining detector 54 and the magnetometer, as well as the IMU of the position sensor 56.Device drivers allow the computer to receive measurement data from associated sensors and send control data to them. In one preferred embodiment, measurement and control data are exchanged between devices and processes running on the computer as messages in the robotic operating system (ROS). Data from the sensors (including the range detector) is received at a rate of 10 Hz, while measurements from the magnetometer and IMU are received at a rate of 100 Hz. Each message is written to the computer's disk.

[0077] The computer 62 outputs control signals to the power control unit 64 and, optionally, receives diagnostic data from the power control unit. The power control unit supplies power via the cable hose 38 to the upper lighting matrix 22 and the lower lighting matrix 24. In the preferred embodiment, the power control unit 64 receives power at 220 V AC, which can be supplied directly to the charger of the capacitor bank for the xenon flash lamps within the upper lighting matrix 22 (when it is started). The power control unit transfers power to the underwater junction box (not shown), which converts the AC power to DC power (for example, 36 V or 72 V) for the LED lamps within the upper lighting matrix 24.

[0078] The focus calculation process performed by computer 62 continuously monitors the ranging data to detect the presence and range of fish within the target area. The ranging data consists of one or more distances for each detector element from which light was reflected back to the detector element from within its detection angle within the detection fan.

[0079] Figure 4 shows a side view of the angular fields of view 66 of the range-determining detector elements in the receiving optical means 60 within the range-determining perception fan 68. As described above, the aperture angles of adjacent detectors are adjacent to each other in pitch, jointly creating a perception fan. Figure 4 shows an array of 16 detectors, each with a field of view whose opening occupies approximately 3° in pitch.

[0080] Figure 4 shows the average installation angles of the detectors within the range-determining fan 68. Each average installation angle is illustrated by a geometric axis 70, which bisects the field of view 66 of the corresponding detector. The average installation angle is the angle between the bisecting geometric axis 70 (bisector) and the geometric axis of the fan 68 as a whole, which is generally parallel to the optical axis of the camera 52.

[0081] Figure 4 also shows a side view of the distances and average installation angles for several detector elements occluded by fish 72, 74 within the range-determining perception fan 68. In general, the focus calculation process ensures the detection of fish when several adjacent detector elements report similar distances. In a preferred embodiment of the invention, a fish is detected when M or more adjacent detector elements report similar distances d iThe number M can range from 1 to 1 / 2 of the total number of detectors (i.e., 8 in this example). In particular, the process of calculating the focus requires searching for adjacent sets of M or more adjacent distances d. i , for which [max(d i )—min(d i )]≤W. M and W are parameters that can be adjusted by the image acquisition system operator, with W representing the maximum allowable thickness, approximately equal to half the thickness of the largest fish that will be detected. Depending on the size or age of the fish, the parameters M and W are optimized for each system or each farm. For each such detection, the focus calculation involves calculating the average distance

[0082]

[0083] and medium bearing

[0084]

[0085] where β i- the average installation angles of each of the adjacent detector elements. The focus calculation process then returns the focal length D f = D × cos β, which is the distance from the camera to the recommended focal plane along the optical axis of the camera.

[0086] Due to scattered particles in the water or an object (e.g., a fish) that only partially obscures a detector's detection angle, a single detector can report multiple distances. In a practical implementation, in cases where a single detector reports multiple distances, the largest distance is used when calculating the focal point. This minimizes false detections caused by particles within the water. If multiple distances are actually associated with two fish, one of which only partially obscures a detector's detection angle, adjacent detectors will likely still successfully detect the partially obscuring fish.

[0087] The presence and range information of fish, determined through the focus calculation process, can be used to control image acquisition by the camera. For example, image capture is only possible when a fish is detected within a predetermined distance of the "sweet spot" that provides optimal lighting. For example, if the "sweet spot" is 700 mm, image capture is only possible when a fish is detected within a certain range of 600 to 800 mm. Each time image capture occurs, the camera and lighting control system sets the camera's focal length to the most recent range value determined through the focus calculation process.

[0088] In one preferred embodiment of the invention, the camera and lighting control system continuously triggers the camera to acquire images periodically, for example, at a frequency of 4 Hz or, more generally, at a frequency between 2 and 10 Hz. The focus calculation process provides for continuous and periodic (for example, at a frequency of 10 Hz or, more generally, at a frequency of 4 to 20 Hz) reporting of the current focal length based on the most recent detected fish ranges, and the camera and lighting control system sets the focal length of the camera to the most recently known focal length.

[0089] When the shutter of the camera opens, the camera sends a synchronization signal to the camera and lighting control system 12, which is passed to the power control unit 64. The power control unit ensures that the illumination of the upper illumination matrix 22 and the lower illumination matrix 24 is synchronized with the release of the shutter, ensuring proper illumination of the captured image. In those embodiments of the invention where the lamps (such as xenon flash lamps according to the preferred embodiment of the invention) within the upper illumination matrix or the lower illumination matrix are not capable of maintaining the same duty cycle as the camera, the power control unit can also provide for a lighting stabilization process that includes continuously evaluating whether the power control unit should ensure the illumination of the upper illumination matrix and the lower illumination matrix.In one preferred embodiment of the invention, the illumination is stabilized if either (a) the history of the xenon flash lamps within the upper illumination array indicates that they are approaching their upper thermal limit, or (b) the focus calculation process has not resulted in a recent fish detection and reporting of an updated range.

[0090] In one preferred embodiment of the invention, lower-intensity LED lamps within the lower illumination array are illuminated for the duration of the camera exposure. The exposure duration is set to the minimum duration required for the LED lamps to provide equivalent illumination. The flash duration of the xenon flash lamps in the upper illumination array is adjusted to provide balanced illumination, taking into account the protective coloration of typical fish.

[0091] The illumination intensity provided by the LED lamps within the lower illumination matrix is ​​preferably high enough to ensure a short enough exposure to produce acceptably low motion blur within the captured images of swimming fish. In a preferred embodiment of the invention, the sensor inside the camera (in particular, its number of pixels), the optics of the camera (in particular, the angular resolution of the field of view), and the distance to the target area are selected to ensure that (a) the entire fish can be captured within the field of view of the camera, and (b) adequate resolution of juvenile external parasites of fish, such as sea lice, is possible.To provide 10 pixels per 2 mm (comparable to the size of juvenile sea lice) at a target distance where the 60° horizontal field of view of the camera covers the width of a typical mature fish, an angular pixel pitch of 7.6×10 is required. −3 ° per pixel. For fish swimming at a typical speed of 0.2 m / sec, sub-pixel motion blur is guaranteed by shutter times of less than 0.6×10 −3 To provide adequate, low-noise images to a machine vision system, a sensor gain of less than ISO 3200 is preferred. This, in turn, requires an illumination of approximately 3000 lux over the entire target area.

[0092] Figure 5 illustrates the results that would be obtained using the range-determining detector 54 in the situation depicted in Figure 4. What is shown are the detection results from detector elements with fields of view having geometric axes in the range from +6° to -15°. Each black dot in Figure 5 represents a detection event where the reflected light is received by the corresponding detector element. The position of the dot in the direction of the d-axis represents the distance to the detected object, calculated based on the travel time of the light signal from the emitting optical means 58 to the object and back to the receiving optical means 60.

[0093] As described previously, fish 72 and 74 are represented by detections at approximately equal distances d1 and d2, respectively, for a number of adjacent detectors. For each individual detector, the distance to the fish is the largest among the distances measured by that detector. Points at smaller distances represent noise caused by small particles in the detection fan.

[0094] In the situation illustrated in Figs. 4 and 5, fish 74 is partially obscured by fish 72, so that the image of the entire fish silhouette can only be obtained for fish 72 located at a shorter distance d1. Therefore, the focus of the camera will be adjusted to obtain that distance d1.

[0095] Figure 6 shows a timing diagram showing the detections at a distance d1 as a function of time t. It can be seen that the obtained detections of fish 72 for angles β in the range of -3° to -15° are stable over a long period of time, corresponding to the time it takes for the fish to swim through the perception fan 68. Therefore, it would also be possible to filter out noise by imposing a requirement that the detection should be stable over a certain minimum time interval or, equivalently, by integrating the signal received from each detector element over a certain time and then adjusting the threshold of the integration result.

[0096] In principle, the detection archive of the type illustrated in Fig. 6 could also be used to optimize the time interval in which the camera 52 acquires a sequence of images to ensure, on the one hand, that the number of images does not become unreasonably large and, on the other hand, that the sequence of images includes at least one image in which all the fish are within the field of view of the camera. For example, as shown in Fig. 6, at time t1, when a certain number of adjacent detector elements (three) detect an object that could be a fish, a certain timer can be started.Then - with some definite delay - at time t2 the camera can be started to begin taking a sequence of pictures, and this sequence will be stopped - at the latest - at time t3, when the detector elements indicate that the end of the fish's tail leaves the perception fan.

[0097] Figure 7 shows the field of view 76 of the filming apparatus at the time t1 shown in Figure 6, when the nose of the fish 72 has just crossed the perception fan 68.

[0098] Figure 8 shows an image captured by the camera 52 at a time slightly later than the time t2 shown in Figure 6, when the entire silhouette of the fish 72 is within the field of view 76. At this time, from the detection results of the detector elements at β in the range from -3° to -15° according to Figure 6, it can be concluded that the geometric axis of the fish will pass at an angle of β = -9°, as shown in Figure 8. This information can be transmitted to the image processing system and can facilitate the recognition of the outline of the fish in the captured image.

[0099] Returning to Fig. 1, we note that the computer 62 of the camera and lighting control system 12 is connected to the image processing system 78, which has access to the database 80 through the data management system 82.

[0100] Data Management System

[0101] The automated system for detecting and counting external parasites of fish such as sea lice also includes a data management system 82 that includes interfaces that support the acquisition, storage, retrieval, retrieval and distribution of image data, image metadata, image annotations and detection data generated during operation of the image processing system 78.

[0102] Data storage

[0103] Data management system 82 receives images from the image capture system, for example, in the form of ROS "packets." The data management system decompresses each packet, for example, into a JPEG or PNG image and JavaScript Object Notation (JSON) metadata. Each JPEG image is stored in the data warehouse.

[0104] Database

[0105] JSON metadata, unpacked from each POC packet, is stored in database 80 associated with the data warehouse. Generally, the metadata describes image capture parameters related to the associated JPEG or PNG image. For example, the metadata includes the centroid pixel cell within the fish silhouette (e.g., the pixel centered horizontally within the image, along the longitudinal geometric axis) detected by the LEDDAR (light-emitting diode detection and ranging) unit. This pixel cell can optionally be used by the image processing system (described in more detail below) to facilitate fish detection within the image.

[0106] Database 80 also stores annotation data generated during the annotation process (described in more detail below) for training image processing system 78. Furthermore, the database stores information characterizing the location, size, and type of fish and external fish parasites, such as sea lice, detected by the machine vision system. Finally, the database stores authentication credentials, allowing users to log in to various interfaces (e.g., the annotation interface or the end-user interface) through the authentication module.

[0107] Image processing system

[0108] In a specific embodiment, the invention provides for the use of an image processing system 78 for detecting external fish parasites such as sea lice. In a preferred embodiment, separate neural networks are trained to provide a fish detector 84 and a fish external parasite detector 86. First, the fish detector 84 detects individual fish within images acquired by the image capture system. Then, the fish external parasite detector 86 detects individual external fish parasites such as sea lice (if present) on the surface of each detected fish. In a preferred embodiment, the fish external parasite detector also classifies the sex and life stage of each detected louse.

[0109] The detectors are trained using a machine learning procedure that assimilates a corpus of human-annotated images. The use of a neural network eliminates the need to explicitly identify characteristics (such as length, shape, brightness, color, or texture) of fish or external fish parasites such as sea lice, and instead draws inferences directly based on the knowledge possessed by human annotators, encoded within the corpus of annotated images.

[0110] Figure 9 shows the position and silhouette of the fish 72 in the field of view 76, detected by the fish detector 84. The other fish 74 shown in Figures 4, 7 and 8 are excluded from consideration in this embodiment, since they are partially obscured by the fish 72. However, in a modified embodiment, it would be possible to also detect the fish 74 and search for an external fish parasite, such as sea lice, on the skin of the fish 74, insofar as it is visible.

[0111] In one embodiment of the invention, the focal depth of the filming apparatus 52 is selected such that a clear image is obtained for the entire silhouette of the fish 72. In a modified embodiment, as shown in Fig. 9, the silhouette of the fish recognized by the fish detector is divided into subzones 88, which differ in their distance from the filming apparatus 52. The distances in different subzones 88 are calculated based on the result of the range determination obtained from the range-determining detector 54. The values ​​of the distances obtained by means of different detector elements of the range-determining detector already reflect the effect of the angular misalignment between the geometric axis 70 of the field of view and the optical axis of the filming apparatus in the trim direction. In addition, for each point within the silhouette of the fish 72, a conclusion about the effect of the angular misalignment in the horizontal direction can be made based on the position of the pixel on the fish in the field of view 76.Optionally, one can make a correction to the distance for the relief of the fish's body in the horizontal direction, and this relief is at least approximately known for the fish species under consideration.

[0112] Then, when a series of images of the fish 72 are taken (for example, at a frequency of 4 Hz, as described above), the focus can be changed from image to image in such a way that a corresponding adaptation of the focus to one of the sub-zones 88 shown in Fig. 9 occurs. This makes it possible to obtain high-resolution images of all sub-zones 88 of the fish with a reduced depth of focus and - accordingly - with a setting of the aperture of the camera, which will require a lower intensity of light during illumination.

[0113] Then, when a series of images of the fish 72 are taken (for example, at a frequency of 4 Hz, as described above), the focus can be changed from image to image in such a way that a corresponding adaptation of the focus to one of the sub-zones 88 shown in Fig. 9 occurs. This makes it possible to obtain high-resolution images of all sub-zones 88 of the fish with a reduced depth of focus and - accordingly - with an aperture setting of the camera that will require less light intensity during illumination.

[0114] Figure 10 shows a normalized image of fish 72, which is subsequently fed to external fish parasite detector 86. This image can optionally be composed of multiple images of subzones 88 captured with different camera focuses. Furthermore, the image shown in Figure 10 can be size-normalized, bringing it to a standard size, facilitating comparison of the captured fish image with annotated images.

[0115] It will be noted that the image of fish 72, recognized and shown in Fig. 9, may be subject to distortion (horizontal compression) if the fish is not oriented at a right angle to the optical axis of the camera. The normalization process, resulting in the fish silhouette shown in Fig. 10, can compensate for this distortion.

[0116] Furthermore, Fig. 10 illustrates an optional embodiment in which the silhouette of the fish is divided into different regions 90, 92, and 94-100. Regions 90 and 92 allow the image processing system to distinguish between the upper side and the lower side, for which, on the one hand, the skin color of the fish will be high-resolution, and on the other hand, the intensities and spectra of the illumination provided by the upper and lower illumination matrices 22 and 24 will be different from each other. Knowing the region 90 or 92 where the pixel is located on the fish simplifies the search for characteristic features by the fish external parasite detector 86 in the contrast between such external fish parasites as sea lice and the fish tissue.

[0117] Additional regions 94-100 shown in this example denote selected anatomical features of fish that correlate with characteristic population densities of external fish parasites, such as sea lice of various species, on the fish. These same anatomical regions 94-100 will also be identified in the annotated images used for machine learning. This allows the fish external parasite detector to be trained or configured such that the confidence levels for detecting external fish parasites, such as sea lice, are adapted to the region being surveyed.

[0118] Furthermore, when the fish external parasite detector 86 is operated in the inference mode, it is possible to provide separate statistics for different areas 94-100 on the fish, which can provide information useful for identifying the species, sex, and / or life stage of such fish external parasites as sea lice, and / or the degree of infestation.

[0119] Annotation, training, validation, and testing

[0120] Figure 11 shows a flow chart detailing the annotation of images and the training, validation, and testing of detectors 84, 86 within the image processing system 78. The annotation and training process begins with image acquisition. The annotation interface 102 enables users to create a set of annotations, which, when associated with relevant images, yields a corpus of annotated images.

[0121] In a preferred embodiment of the invention, the annotation interface communicates with a media server connected to a data store in a database 80. The annotation interface may be based on Hypertext Markup Language (HTML), allowing human annotators to upload, view, and annotate images within a web browser. For each image, the annotator creates a polygon containing each clearly visible fish and a bounding box containing any external fish parasites, such as sea lice, present on the fish's surface. Preferably, the annotation interface also allows the annotator to create bounding boxes containing the fish's eyes (which may be visually similar to external fish parasites, such as sea lice). Preferably, the annotator also indicates the species, sex, and life stage of each louse.

[0122] Annotations created using the 102 annotation interface are stored in the 80 database. When entered into and retrieved from the database, annotations for a single image are serialized as a JSON object with a pointer to the associated image. This simplifies the ingestion of the annotated corpus by machine learning.

[0123] In a preferred embodiment of the invention, the machine learning procedure involves training neural networks on a corpus of annotated images. As shown in Fig. 11, the annotated images can be divided into three image sets. The first two image sets are used for training and validating the neural network. Specifically, the first image set (e.g., approximately 80% of the annotated images) is used to iteratively adjust the weights within the neural network. Periodically (i.e., after a certain number of additional iterations), the second image set (approximately 10% of the annotated images) is used to validate an evolution detector that protects against overfitting. The result of the training and simultaneous validation process is a trained detector 84, 86.A third set of images (e.g., approximately 10% of the annotated images) is used to test the trained detector. The testing procedure characterizes the performance of the trained detector, resulting in a set of performance metrics.

[0124] As shown in Fig. 11, the entire process of training, validation, and testing can be iterated several times as part of a broader neural network design process until acceptable performance metrics are achieved. As noted above, in the preferred embodiment of the invention, the process according to Fig. 11 is performed at least once to create a fish detector 84, and at least once to create a fish external parasite detector 86.

[0125] In alternative embodiments of the invention, to improve the quality of the training, validation, and testing process, the machine learning procedure includes a data augmentation process to increase the size of the annotated corpus. For example, the use of augmentation methods such as noise accumulation and perspective transformation of the human-annotated corpus can increase the size of the training corpus by 64 times.

Claims

1. A method for monitoring external parasites of fish in aquaculture, in which: the filming apparatus (52) is immersed in a sea farm (40) containing fish (72, 74), wherein the filming apparatus has a field of view; capture images of fish (72, 74) using a camera (52); and identify an external fish parasite on a fish (72, 74) by analyzing captured images, characterized in that illuminate the target area in the field of view of the camera (52) from above and below; in this case, an electronic image processing system (78) is used to detect fish (72, 74) in the image captured by the camera (52); and additionally: train a neural network to identify an external parasite of fish and use the trained neural network in an electronic image processing system (78); operate the range-determining detector (54) to continuously monitor the portion of the sea farm (40) to detect the presence of fish in that portion of the sea farm and, when a fish is detected, measure the distance from the survey apparatus (52) to the fish (72, 74); and, when the fish is detected, calculating the focusing settings of the camera (52) based on the measured distance; and starting the camera (52) when the detected fish (72, 74) is within a predetermined distance range; wherein the range detector (54) is used to measure the bearing angle of the detected fish (72), and the measured bearing angle is used in the image processing system (78) to search for the silhouette of the fish in the captured image.

2. The method according to paragraph 1, characterized in that the target area has dimensions sufficient to accommodate the silhouette of the fish in its entirety.

3. The method according to claim 1, characterized in that the target area in the field of view of the filming apparatus (52) is illuminated from above and below with light of varying intensity and / or different spectral composition.

4. The method according to paragraph 1, characterized in that when identifying an external parasite of a fish, the parasites are identified in a given location on the fish.

5. The method according to claim 1, characterized in that when identifying an external parasite of a fish, a distinction is made as to whether the specified location on the fish is an area (90) of the upper side or an area (92) of the lower side of the fish.

6. A system for monitoring external parasites of fish in aquaculture, comprising: filming equipment (52); lighting equipment (10) that illuminates the target area from above and below; an electronic image processing system (78) for detecting fish (72, 74) in an image captured by a camera (52) and for detecting external fish parasites in the silhouette of the detected fish, wherein the electronic image processing system is configured to use a trained neural network for detecting external fish parasites; a range-determining detector (54) configured to continuously monitor a portion of the sea farm (40) to detect the presence of fish in that portion of the sea farm and, when a fish is detected, to measure the distance from the filming apparatus (52) to the fish (72, 74), and when a fish is detected, to calculate a focus setting of the filming apparatus (52) based on the measured distance; and to trigger the filming apparatus (52) when the detected fish (72, 74) is within a predetermined distance range; wherein the range-determining detector (54) is configured to be used to measure the bearing angle of the detected fish (72), and the measured bearing angle is used in the image processing system (78) to search for the silhouette of the fish in the captured image.

7. The system of claim 6, wherein the light from above the target area has a different intensity and / or a different spectral composition than the intensity and / or spectral composition of the light from below the target area.

8. The system according to claim 6, further comprising a position sensor (56).

9. The system of claim 6, wherein the trained neural network is configured to detect an external parasite of a fish at a given location on the fish.

Images