Estimating fish respiration activity in an aquaculture environment

Abstract

An automated method for monitoring the breathing rate of freely swimming fish in an aquaculture environment. The method utilizes computer vision techniques to track individual fish across video frames captured by an underwater camera. A trained machine learning classifier analyzes the mouth states of the fish in each frame, classifying them as open or closed. The method aggregates the mouth state data from multiple fish tracks to determine the overall mouth state transition count, which is used to calculate the average breathing rate. The average breathing rate is then outputted to a graphical user interface, database, or report. This automated approach offers a non-invasive, efficient, and reliable means of assessing the respiratory health of fish populations in aquaculture settings, reducing the reliance on manual observation and enabling continuous monitoring. The method contributes to the optimization of fish welfare and the sustainability of the aquaculture industry.

Description

ESTIMATING FISH RESPIRATION ACTIVITY IN AN AQUACULTUREENVIRONMENTBACKGROUND

[0001] Fish aquaculture, also known as fish farming, is the practice of raising fish in controlled environments for commercial purposes. This method of food production involves cultivating freshwater or saltwater fish species in enclosed spaces such as ponds, tanks, or cages, where they are fed and cared for until they reach a marketable size. Aquaculture has become increasingly important in meeting the global demand for fish and seafood, as wild fish populations have been declining due to overfishing and environmental factors. Fish farming allows for a more consistent and predictable supply of fish, while also reducing pressure on wild fish stocks.

[0002] An experiment conducted by SINTEF of Norway was designed to study the effects of increased carbon dioxide levels on fish respiration, a land-based tank is used to house the fish while carbon dioxide is pumped into the water. The behavior of the fish is monitored using a camera system, which allows researchers to observe the mouth openings and closings of the fish, serving as an indicator of their respiratory activity. The results of the SINTEF experiment reveal that fish exhibit higher breathing frequencies when exposed to elevated carbon dioxide concentrations in the water. The observed frequency of mouth openings ranges from 0.7 to 1.6 Hz, indicating a significant increase in respiratory activity compared to normal conditions. This finding suggests that fish are actively compensating for the increased carbon dioxide levels by increasing their ventilation rate to maintain adequate oxygen uptake.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The following detailed description of certain embodiments of the invention are understood by reference to the following figures:

[0004] FIG. 1 illustrates an example aquaculture environment in which techniques for estimating fish respiratory activity are implemented, according to an embodiment.

[0005] FIG. 2 illustrates examples of classified tracks, according to an embodiment.

[0006] FIG. 3 illustrates an example of track splitting, according to an embodiment.

[0007] FIG. 4 illustrates a method for estimating fish respiratory activity in an aquaculture environment, according to an embodiment.

[0008] FIG. 5 illustrates an example of a programmable electronic device that processes and manipulates data to perform the techniques disclosed herein for estimating fish respiratory activity in an aquaculture environment, according to an embodiment.DETAILED DESCRIPTION

[0009] Systems, methods, and non-transitory computer-readable media (collectively referred to as “techniques”) are disclosed for estimating fish respiratory activity in an aquaculture environment.

[0010] An embodiment of the techniques encompasses an innovative approach to monitoring the health and well-being of fish in an aquaculture environment. It involves using advanced computer vision techniques to analyze video footage captured by an underwater camera that is immersed in the aquaculture environment where the fish are freely swimming. The computer vision algorithm tracks individual fish across a series of video frames, creating a set of tracks where each track represents a specific fish and comprises a subset of time-ordered video frames.

[0011] For each track, a trained machine learning classifier is applied to the corresponding subset of time-ordered video frames. This classifier is designed to determine the mouth state of the fish in each frame, such as whether the mouth is open or closed. By analyzing the mouth states over time, the classifier produces a set of time-ordered fish mouth states for each track.

[0012] The sets of time-ordered fish mouth states from all the tracks are then aggregated to determine an overall mouth state transition count. This count represents the total number of times the fish in the aquaculture environment transition between different mouth states, such as from open to closed or vice versa. By calculating this transition count, the method can estimate the average breathing rate of the fish population.

[0013] Finally, the determined average breathing rate is outputted to a graphical user interface, stored in a database, or included in a report. This information can be valuable for aquaculture managers and researchers in assessing the health and stress levels of the fish, as changes in breathing rate can be indicative of various environmental factors or health issues. By using this method, aquaculture professionals can monitor the well-being of their fish population in a non- invasive manner and make informed decisions to optimize the growth and quality of the fish.

[0014] FISH RESPIRATORY PROCESSES

[0015] Fish, such as salmon, possess a unique respiratory system that allows them to extract oxygen dissolved in water using their gills. This process is essential for their survival and is commonly referred to as "buccal pumping." During buccal pumping, fish actively draw water into their mouth and push it over their gills, where the exchange of gases takes place. The gills are composed of thin, feathery structures called gill filaments, which are rich in blood vessels.

[0016] As water passes over these filaments, the dissolved oxygen in the water diffuses into the fish's bloodstream, while carbon dioxide is simultaneously released from the blood into the water. This gas exchange process occurs at a relatively consistent frequency, typically ranging from 0.5 to 2 Hz (cycles per second), depending on the species and environmental conditions.The rate of buccal pumping can be influenced by factors such as water temperature, oxygen availability, and the fish's activity level. By maintaining a steady rhythm of buccal pumping, fish ensure that their bodies receive a constant supply of oxygen to support their metabolic processes and overall health.

[0017] The sequence of actions described above pertains to the buccal pumping behavior exhibited by fish, which is part of their respiration process. The first step involves the simultaneous opening of the fish's mouth and the oral valve, a specialized organ located within the oral cavity. This coordinated action allows the fish to actively draw water into its mouth and oral cavity.

[0018] Once the water is taken in, the fish closes its mouth along with the oral valve. It is important to note that although the mouth may appear slightly open at this stage, the oral valve acts as a barrier to prevent water from flowing back out through the mouth. This valve ensures that the water is directed towards the gills for the next stage of the respiration process.

[0019] Following the closure of the mouth and oral valve, the fish proceeds to open its operculum, which is a bony flap that covers and protects the gills. The operculum's movement exposes the gills to the water that was previously taken into the oral cavity.

[0020] Finally, the fish exhales the water through its gills. As the water passes over the gill filaments, the exchange of gases takes place. The dissolved oxygen in the water is absorbed into the fish's bloodstream, while carbon dioxide is simultaneously released from the blood into the water. This process allows the fish to obtain the necessary oxygen for its metabolic processes and eliminate the carbon dioxide produced as a waste product.

[0021] By repeatedly executing this sequence of actions, fish can efficiently extract oxygen from the water and maintain proper respiration, enabling them to thrive in their aquatic environment.

[0022] When fish have trouble breathing or are subjected to increased metabolic demands, they adapt by modifying their ventilation rate to compensate for the additional oxygen requirements. This change in ventilation frequency is manifested through two observable behaviors: an increase in the frequency of mouth openings, known as the gape rate, and an increase in the movement of the operculum, referred to as the operculum rate.

[0023] The gape rate refers to the number of times a fish opens and closes its mouth within a given time period. When a fish is struggling to breathe or is engaging in activities that require higher metabolic function, such as swimming vigorously or digesting a meal, it will typically increase its gape rate. By opening and closing its mouth more frequently, the fish can draw in a greater volume of water over its gills, thereby enhancing the uptake of oxygen from the water.

[0024] Similarly, the operculum rate, which is the frequency at which the fish's operculum (gill cover) moves, also increases under these circumstances. The operculum facilitates the respiration process by protecting the gills and facilitating the flow of water over them. During periods of increased metabolic demand or respiratory stress, fish will exhibit a higher operculum rate, indicating that they are actively pumping water over their gills at a faster pace to maximize oxygen extraction.

[0025] The combination of increased gape rate and operculum rate allows fish to efficiently meet their elevated oxygen requirements during challenging situations. By monitoring these behavioral indicators, researchers and aquaculture professionals can gain valuable insights into the respiratory health and stress levels of fish, enabling them to take appropriate measures to ensure the well-being of the fish under their care.

[0026] In situations where fish, such as salmon, require additional oxygen to meet their metabolic needs, they may transition from the standard buccal pumping mode of respiration to a different mode called "ram ventilation." This adaptation is particularly useful when the fish is experiencing high levels of stress, engaging in intense physical activity, or encountering low oxygen concentrations in the water.

[0027] During ram ventilation, the fish swims forward with its mouth and operculum (gill cover) constantly held open. This behavior allows a continuous flow of water to pass over the fish's gills, maximizing the surface area available for gas exchange. By maintaining a steady forward motion, the fish creates a pressure difference between the front and back of its body, which drives the water through its open mouth and over the gills.

[0028] The continuous flow of water during ram ventilation enables the fish to extract oxygen more efficiently compared to the intermittent water flow generated by buccal pumping. This increased efficiency is particularly advantageous when the fish's oxygen demands are high, as it allows them to obtain the necessary oxygen to support their elevated metabolic requirements.

[0029] However, ram ventilation also comes with certain trade-offs. Swimming with an open mouth and operculum can increase drag, making it more energetically demanding for the fish to maintain its forward motion. Additionally, the constant exposure of the gills to the water can make the fish more susceptible to irritants, pathogens, or other harmful substances present in the environment.

[0030] Despite these challenges, ram ventilation remains an important respiratory adaptation for fish like salmon, allowing them to cope with situations that require enhanced oxygen uptake. By understanding and monitoring the occurrence of ram ventilation using techniques disclosed herein, researchers and fish managers can gain valuable insights into the respiratory health and stress levels of fish in various aquatic environments.

[0031] Fish, like all living organisms, require a sufficient supply of oxygen to maintain their bodily functions and overall health. However, various environmental and biological factors can lead to increased ventilation in fish, which is characterized by a higher frequency of mouth openings and operculum movements. One of the primary reasons for increased ventilation is low oxygen levels in the water. When the dissolved oxygen concentration in the water is inadequate, fish need to pump more water over their gills to extract the necessary amount of oxygen, resulting in increased ventilation.

[0032] Gill issues can also contribute to increased ventilation in fish. The gills are the primary organ responsible for gas exchange, and any damage or obstruction to the gills can impair their ability to efficiently extract oxygen from the water. Pathogens, such as string jellyfish and amoebic gill disease, can cause inflammation, lesions, or excessive mucus production in the gills, hindering their normal function and leading to increased ventilation as the fish attempts to compensate for the reduced oxygen uptake.

[0033] Moreover, heart problems can also result in increased ventilation in fish. The heart plays an important role in pumping oxygenated blood throughout the fish's body. If the heart is not functioning optimally, it can lead to reduced oxygen delivery to the tissues, prompting the fish to increase its ventilation rate to obtain more oxygen.

[0034] Environmental factors, such as strong currents, can also contribute to increased ventilation in fish. Swimming against strong currents requires more energy and oxygen consumption, leading to a higher ventilation rate to meet the increased metabolic demands.

[0035] In some cases, a combination of these factors may be responsible for the increased ventilation observed in fish. For example, low oxygen levels in the water coupled with gill issues can exacerbate the problem, leading to even higher ventilation rates as the fish struggles to obtain sufficient oxygen.

[0036] Understanding the various factors that can cause increased ventilation in fish is useful for maintaining their health and well-being in both natural and aquaculture settings. By monitoring ventilation rates using the techniques disclosed herein and identifying the underlying causes, fish managers and researchers can take appropriate measures to mitigate the impact of these factors and ensure the optimal growth and survival of the fish under their care. Furthermore, applying a calibrated model to respiration activity can also yield an estimate of metabolic load, which is informative for estimating the growth and feed conversion yield for farmed fish.

[0037] RESPIRATORY INDEX

[0038] A simple method for estimating fish respiration activity in an aquaculture environment involves counting the number of images of fish with their mouths open over a specified period. This approach is based on the principle that fish open their mouths to draw in water forrespiration, and the frequency of mouth openings can serve as an indicator of their respiratory activity.

[0039] To implement this method, a camera system is set up to capture images of the fish at regular intervals within the aquaculture environment. The duration of the monitoring period is determined based on the specific requirements of the study or the aquaculture management practices. Once the images are collected, they are analyzed to determine the number of fish with open mouths in each image.

[0040] The analysis process can be conducted manually by a human observer or using computer vision techniques. In the manual approach, the observer examines each image and counts the number of fish with open mouths. This method can be time-consuming and subject to human error, especially when dealing with many images or a high density of fish.

[0041] Alternatively, computer vision algorithms can be employed to automate the counting process. These algorithms are trained to detect and recognize the open mouths of fish in the images. By applying the trained algorithm to the collected images, the system can automatically count the number of fish with open mouths in each image, providing a more efficient and consistent analysis.

[0042] Once the count of fish with open mouths is obtained for each image, the data can be aggregated over the entire monitoring period. This aggregation involves summing up the counts from all the images to determine the total number of instances where fish were observed with open mouths. By dividing this total count by the number of images analyzed, an average frequency of mouth openings (or equivalently a “respiratory index”) can be calculated.

[0043] The respiratory index serves as an estimate of the fish respiration activity in the aquaculture environment. Higher indexes indicate increased respiratory activity, which may be associated with factors such as stress, disease, or suboptimal water conditions. Lower indexes suggest a more relaxed and healthy state of the fish.

[0044] In an experiment (termed “the SINTEF experiment” herein) designed to study the effects of increased carbon dioxide levels on fish respiration, a land-based tank is used to house the fish while carbon dioxide is pumped into the water. The behavior of the fish is monitored using a camera system, which allows researchers to observe the mouth openings and closings of the fish, serving as an indicator of their respiratory activity. The average frequency of mouth openings, referred to herein as the "respiratory index," can be used to quantify the fish's respiratory response to the changing environmental conditions.

[0045] The results of the SINTEF experiment reveal that fish exhibit higher breathing frequencies when exposed to elevated carbon dioxide concentrations in the water. The observed frequency of mouth openings ranges from 0.7 to 1.6 Hz, indicating a significant increase inrespiratory activity compared to normal conditions. This finding suggests that fish are actively compensating for the increased carbon dioxide levels by increasing their ventilation rate to maintain adequate oxygen uptake.

[0046] Based on these experimental results, it is expected that the respiratory index would be able to detect stress in fish that primarily rely on buccal pumping for respiration, up to a certain level of stress. Buccal pumping involves the fish actively drawing water into the mouth and pushing it over the gills for gas exchange. As stress levels increase, such as in the case of elevated carbon dioxide concentrations, fish engaged in buccal pumping will likely exhibit a higher respiratory index, reflecting their increased respiratory effort.

[0047] However, it is important to note that the respiratory index may have limitations in detecting stress beyond a certain threshold. If the stress levels become too high, fish may transition from buccal pumping to other modes of respiration, such as ram ventilation, where they swim forward with their mouth and operculum constantly open to facilitate continuous water flow over the gills. In such cases, the respiratory index, which relies on the frequency of mouth openings, may lose its sensitivity to detect further increases in stress.

[0048] Moreover, the relationship between stress levels and the respiratory index may not be linear, and there may be individual variations among fish in their respiratory response to stress. Some fish may have a higher tolerance to stress and exhibit a less pronounced increase in the respiratory index, while others may be more sensitive and show a more substantial response.

[0049] The respiratory index, which is based on the average frequency of mouth openings, can be an accurate indicator of moderate levels of stress in fish. Under normal conditions, fish exhibit a breathing pattern characterized by a longer duration of mouth closure compared to mouth opening. This means that during a typical breathing cycle, the fish's mouth remains closed for a larger portion of the time.

[0050] Experimental data supports this observation, showing that at low gaping frequencies, the duration of mouth opening (represented by the respiratory index) accounts for approximately 20- 30% of the total time. For example, assuming an average mouth opening duration of 0.5 seconds and a respiratory index of 25%, it can be estimated that the breathing rate is around 0.5 Hz (25% divided by 0.5 seconds).

[0051] However, when fish are subjected to moderate levels of stress, such as increased carbon dioxide concentrations in the water, their breathing patterns begin to change. The duration of mouth closure decreases, while the average duration of mouth opening remains relatively constant. This pattern holds true up to a respiratory index of about 50%.

[0052] This observation aligns with an assumption that the respiratory index can be used to estimate the gaping frequency by dividing the respiratory index by the fixed duration of mouthopening. In the given example, assuming a mouth opening duration of 0.5 seconds and a respiratory index of 50%, the estimated gaping frequency would be 1 Hz (50% divided by 0.5 seconds).

[0053] However, it is important to acknowledge that the accuracy of the respiratory index may diminish beyond the 50% threshold. As stress levels continue to increase, the breathing patterns of fish may undergo further changes, and an assumption of a constant mouth opening duration may no longer hold true.

[0054] As the level of stress on fish increases beyond a certain point, such as when even more carbon dioxide is introduced into the water, the respiratory index may no longer accurately capture the true extent of the fish's respiratory distress. This limitation arises due to changes in the fish's breathing patterns and the breakdown of the assumptions underlying the respiratory index.

[0055] At high stress levels, the gaping frequency of the fish continues to increase, indicating a more rapid and frequent opening and closing of the mouth. However, the respiratory index, which measures the percentage of time the mouth is open, is likely to plateau at around 50%. This means that even as the fish's breathing rate intensifies, the respiratory index may not reflect this increase accurately.

[0056] Furthermore, an assumption of a fixed duration of mouth opening no longer holds true under extreme stress conditions. As the fish's breathing becomes more rapid and erratic, the duration of mouth opening may vary significantly, making it challenging to establish a reliable relationship between the respiratory index and the actual gaping frequency.

[0057] It is speculated that this transition, where the respiratory index loses its ability to accurately capture the gaping frequency, occurs around a gaping rate of 1 Hz. Beyond this point, any further increase in the gaping frequency would likely result in an approximately 50% respiratory index, regardless of the actual breathing rate.

[0058] Interestingly, in some cases, the respiratory index may even appear to decrease paradoxically at very fast breathing rates. This observation suggests that the fish's mouth may be open for a shorter duration during each breathing cycle, despite the overall increase in the gaping frequency. Such a scenario would be problematic for using the respiratory index as a reliable indicator of stress, as it would fail to capture the true extent of the fish's respiratory distress.

[0059] These findings raise concerns about the adequacy of the respiratory index as a standalone measure for assessing extreme respiratory distress in fish that primarily rely on buccal pumping for respiration. While the respiratory index remains a useful tool for monitoring moderate levels of stress, its limitations become apparent when fish experience severe respiratory challenges.

[0060] While estimating high gaping rate frequencies may be challenging for the respiratory index, it does, however, serve as an indicator for ram ventilation behavior. As fish transitions from buccal pumping to ram ventilation due to stress, the respiratory index may increase to well above 50%, which becomes a signal that ram ventilation is taking place in the fish population.

[0061] DETECTING HIGH BUCCAL PUMPING FREQUENCIES

[0062] To accurately detect high buccal pumping frequencies in fish, the inventors realized that it is useful to monitor the opening and closing of the mouth at an individual level. This realization stems from the limitations of relying solely on the respiratory index, especially when fish experience extreme respiratory distress.

[0063] Monitoring mouth movements at an individual level requires tracking a single fish over multiple frames in a video sequence. This approach allows for a more precise measurement of the duration and frequency of mouth opening and closing events. However, this task becomes significantly more complex compared to the simple method of calculating the respiratory index based on the percentage of fish with open mouths in a single frame.

[0064] To efficiently handle this complexity, it may be necessary to develop an automated model that can be deployed. This model would be trained to detect and track individual fish, as well as identify the state of their mouths (open or closed) in each frame. By processing the video stream in real-time, the automated model can provide continuous monitoring of the mouth movements for each fish, enabling the calculation of the buccal pumping frequency.

[0065] An alternative approach could involve manually annotating multiple cropped images uploaded per track. In this case, human annotators would review the images and label the state of the fish's mouth in each frame. However, this manual annotation process can be timeconsuming and costly, especially when dealing with many fish and frames. Additionally, the bandwidth and storage requirements for uploading and storing the cropped images may be prohibitive, making this approach less feasible for large-scale monitoring.

[0066] A third approach could be to manually annotate or use a computer model to automatically detect key points in the head of the fish, labeling features such as the tip of the upper lip, the tip of lower lip, and the base of the jaw. From the distance between the tip of the upper lip and the tip of the lower lip, normalized by the distance of another anatomical feature that does not vary (e.g. base of the jaw to the tip of the upper lip), a threshold can determine if an image yields an open mouth or a closed mouth.

[0067] Another challenge that arises when measuring the mouth opening frequency is the visibility of the fish's mouth throughout an entire open / close / open cycle. In many cases, the fish's orientation or position may obstruct the view of the mouth, making it difficult to directly observe a complete cycle.

[0068] FIRST APPROACH TO ESTIMATING GAPING FREQUENCY

[0069] When attempting to calculate the gape frequency of fish using the respiratory index, the initial approach relied on assuming a fixed duration for mouth opening based on experimental data or using durations from a lookup table. However, as mentioned, the mouth opening duration can vary depending on the stress levels experienced by the fish. This finding challenges the original assumption and highlights the need for a more accurate method to determine the average mouth opening duration.

[0070] In an embodiment, instead of relying on assumptions or predetermined values, a better approach is to directly measure the average duration the mouths are open for a given pen, alongside the respiratory index. Measuring the average mouth opening duration is more feasible than capturing a complete mouth opening and closing cycle for individual fish. This is because the mouth is expected to be open for a very brief period (<0.5 seconds) during buccal pumping, which is just long enough for the fish to be captured in a single frame.

[0071] To implement this approach, live data from the camera system can be utilized. By analyzing the video stream in real-time, the system can identify the moments when a fish's mouth is open and measure the duration of each opening event. These individual duration measurements can then be aggregated to calculate the average mouth opening duration for the entire pen.

[0072] The process of measuring the average mouth opening duration can be automated using computer vision techniques. Algorithms can be developed to detect and track the fish's mouth in each frame, recording the start and end times of each opening event. By accumulating these measurements over a specified period, the system can compute the average duration the mouths are open for the pen.

[0073] Once the average mouth opening duration is obtained, it can be used in conjunction with the respiratory index to calculate the gaping frequency. The same formula proposed above can be applied, where the gaping frequency is estimated by dividing the respiratory index by the average mouth opening duration.

[0074] For example, if the respiratory index is 30% and the measured average mouth opening duration is 0.4 seconds, the estimated gaping frequency would be 0.3 / 0.4 = 0.75 Hz. This calculation provides a more accurate estimation of the gaping frequency compared to using assumed or fixed durations.

[0075] By directly measuring the average mouth opening duration from live data, the method accounts for the dynamic nature of the fish's respiratory behavior under different stress levels. This first approach eliminates the need for assumptions and ensures that the calculated gaping frequency reflects the actual respiratory patterns observed in the pen.

[0076] However, it is important to note that measuring the average mouth opening duration requires a reliable and accurate computer vision system capable of detecting and tracking the fish's mouth in real-time. The system should be able to handle variations in fish orientation, lighting conditions, and other environmental factors that may affect the visibility of the mouth.

[0077] SECOND APPROACH TO ESTIMATING GAPING FREQUENCY

[0078] To estimate the gaping frequency in fish, an alternative approach involves counting the transitions between mouth opening and closing across all observations and tracks. This method relies on aggregating data from multiple fish and calculating the total duration of the observed data.

[0079] The process begins by identifying the moments when a fish's mouth transitions from a closed state to an open state and vice versa. These transitions are counted across all the available tracks, regardless of whether a complete opening and closing cycle is observed for an individual fish. By focusing on the transitions rather than the complete cycles, this approach maximizes the utilization of the available data.

[0080] Once the count of mouth opening to closing transitions and mouth closing to opening transitions is obtained, the average gaping frequency can be calculated using a formula. The formula divides the sum of the transition counts by twice the total duration of the tracks. This calculation effectively estimates the average number of gaping events per unit time.

[0081] The beauty of this second approach lies in its sample efficiency. By aggregating data from all tracks with reliable mouth observations, even if no complete opening and closing cycles are observed, the method maximizes the use of the available information. This is particularly advantageous when dealing with challenging conditions where the mouth may not be visible for extended periods.

[0082] Moreover, this approach is robust to measurement noise. In cases where the mouth state is uncertain, such as when it appears semi -open or is occluded, those observations can be ignored. The focus is solely on the transitions between definitively open and definitively closed states, minimizing the impact of ambiguous or noisy detections.

[0083] It is important to consider the capture rate of the camera system when implementing this method. The current capture rate of 8 Hz is expected to be sufficient for measuring the gape rate, given that the maximum expected gape rate is around 2 Hz. According to the Nyquist- Shannon sampling theorem, the sampling frequency should be at least twice the highest frequency component of the signal to avoid aliasing. With a maximum gape rate of 2 Hz, the Nyquist frequency is 4 Hz, providing a comfortable margin for accommodating some noise in the measurements.

[0084] AUTOMATED MODEL CONSIDERATIONS

[0085] When developing an automated model for detecting mouth openings and closings in fish, several important considerations are taken into account. One of the key challenges is the presence of the oral valve, which can complicate the accurate identification of the mouth state.

[0086] The oral valve is a structure located within the fish's mouth that regulates water flow during respiration. When the mouth is closed, the oral valve prevents water from escaping, allowing the fish to efficiently pump water over its gills. However, the oral valve can sometimes create ambiguity in determining whether the mouth is fully closed or slightly open.

[0087] To address this challenge, one approach is to design a three-class machine learning classifier for the automated model. In an embodiment, a machine learning classifier is trained to categorize the mouth state into three distinct categories in addition to an “unknown,” “occluded,” or “uncertain” category:

[0088] Mouth / oral valve definitely closed: This category represents instances where both the mouth and the oral valve are completely closed, indicating a clear absence of water intake.

[0089] Mouth semi -open: This category captures situations where the mouth appears to be slightly open, but the oral valve may still be partially or fully engaged. In these cases, the model acknowledges the uncertainty in determining the exact state of the mouth.

[0090] Mouth definitely open: This category corresponds to instances where the mouth is unambiguously open, allowing water to flow freely into the oral cavity.

[0091] By incorporating a semi-open category, the model can account for the ambiguity introduced by the oral valve and provide a more nuanced assessment of the mouth state.

[0092] In addition to the three-class classifier, the automated model should also consider cases where the mouth is occluded, or the state is uncertain. Occlusions can occur when the fish's orientation or position in the video frame obscures the view of the mouth, making it difficult to determine its state accurately. Similarly, there may be instances where the model cannot confidently classify the mouth state due to factors such as image quality, lighting conditions, or the presence of foreign objects.

[0093] To handle these situations, the model should include a mechanism for flagging occluded or uncertain mouth states. By explicitly identifying these cases, the model can avoid making unreliable predictions and ensure that only confident assessments are used for further analysis.

[0094] Furthermore, when counting transitions between mouth states, the model should enforce a strict criterion: a transition must involve going from a definitely closed state to a definitely open state. This requirement helps to mitigate the impact of uncertain or ambiguous observations on the overall gaping frequency estimation.

[0095] By implementing a three-class machine learning classifier, accounting for occlusions and uncertain states, and enforcing a strict transition criterion, the automated model can provide a more robust and reliable assessment of mouth openings and closings in fish. These considerations aim to address the challenges posed by the oral valve and ensure that the model's predictions accurately reflect the true respiratory behavior of the fish.

[0096] However, it is important to note that developing an automated model with these capabilities can require a significant amount of training data, including a diverse range of mouth states and scenarios. The classifier should be trained on a large dataset that captures variations in fish species, environmental conditions, and imaging setups to ensure its generalizability and accuracy.

[0097] ORAL VALVE CONSIDERATIONS

[0098] When analyzing the respiratory behavior of fish, the role of the oral valve in determining the true state of the mouth can be considered. The oral valve is an important anatomical feature that can significantly influence the interpretation of mouth openings and closings.

[0099] The oral valve is a specialized organ located within the fish's mouth cavity. Its primary function is to regulate the flow of water during the respiration process. When the fish engages in buccal pumping, the oral valve acts as a barrier, preventing water from escaping the mouth even when it appears to be slightly open.

[0100] In a captured videoframe, the fish's mouth may seem to be partially open at first glance. However, upon closer examination, it is essential to recognize that the presence of the oral valve can obstruct the water flow, effectively rendering the mouth closed from a functional perspective. Despite the slight opening, the oral valve maintains a tight seal, ensuring that water is efficiently directed over the gills for oxygen exchange.

[0101] Consequently, when assessing the gaping behavior of fish, it may be necessary to consider the state of the oral valve rather than relying solely on the visual appearance of the mouth. A mouth that appears slightly open but is accompanied by an engaged oral valve should be treated as functionally closed for the purpose of analyzing respiratory patterns.

[0102] This distinction is particularly important when developing automated models or conducting manual observations of fish respiration. Failing to account for the oral valve's role can lead to inaccurate interpretations and misclassifications of mouth states. A fish with a seemingly open mouth may be in a closed state due to the oral valve's obstruction of water flow.

[0103] To accurately assess the gaping behavior, observers and automated systems can be trained to recognize the presence and function of the oral valve.

[0104] AQUACULTURE ENVIRONMENT FOR ESTIMATING FISH RESPIRATION ACTIVITY

[0105] FIG. 1 illustrates an example aquaculture environment 100 for estimating fish respiration activity, according to an embodiment.

[0106] In the aquaculture environment 100, fish 102 are raised in a fish farming enclosure 104 where they can swim freely. This enclosure 104 is situated at least partially underwater, such as within a larger body of water such as a lake, sea, or ocean. The specific type of fish 102 being farmed can be one or more of various species such as salmon, trout, tilapia, or other commercially valuable fish.

[0107] To monitor the fish 102 in the enclosure 104, a camera 106 is immersed underwater. This camera 106 captures video footage of the fish 102 as they swim and interact within their environment. The video frames collected by the camera serve as the basis for analyzing the behavior and health of the fish population.

[0108] By using computer vision techniques, an automated model 112 processes the video footage to track individual fish across multiple frames. This tracking process results in a set of tracks, where each track represents the movement of a specific fish over time. The model 108 organizes the video frames associated with each track in chronological order, creating a time- ordered sequence of frames for each tracked fish.

[0109] In the aquaculture environment 100, the fish 102 are not confined to small, individual cages or compartments, but rather have the ability to move around and interact with each other in a more natural setting. The free-swimming behavior of the fish 102 is an important aspect of the environment, as it allows for the observation of their natural respiratory patterns and social interactions.

[0110] The aquaculture environment 100 represents a controlled fish farming setup where advanced technology, such as underwater cameras and automated modeling, is employed to monitor and assess the health and well-being of the fish population. This environment 100 enables the collection of valuable data on fish behavior and physiology, which can be used to optimize aquaculture practices and ensure the sustainable production of fish for human consumption.[OHl] In the environment 100, the fish farming enclosure 104 contains freely swimming fish 102. This means that the fish 102 are not restricted to individual cages or compartments but have the ability to move about and interact with each other within the larger enclosure. The free- swimming nature of the fish 102 allows them to exhibit their natural behavior and social dynamics, which is crucial for their overall health and well-being.

[0112] In one embodiment, the fish 102 being raised in the aquaculture environment 100 are Atlantic salmon. Atlantic salmon is a highly valued species in the aquaculture industry due to its rich flavor, nutritional benefits, and high market demand. Farming Atlantic salmon in a free- swimming environment aims to replicate their natural habitat, promoting healthier growth and reducing stress levels compared to more confined farming methods.

[0113] Monitoring the respiratory rate of the freely swimming Atlantic salmon is useful for both their health and harvesting purposes. The respiratory rate, which is the number of breaths taken by a fish per unit of time, serves as a key indicator of the fish's physiological state and overall well-being. By continuously tracking the respiratory rate, aquaculture managers can detect any deviations from normal breathing patterns, which may signify potential health issues or stress within the population.

[0114] Abnormal respiratory rates can indicate a range of problems, such as poor water quality, disease outbreaks, or suboptimal environmental conditions. By promptly identifying these issues through respiratory monitoring, aquaculture managers can take timely corrective actions, such as adjusting water parameters, administering treatments, or implementing quarantine measures. This proactive approach helps prevent the spread of diseases, reduces mortality rates, and ultimately improves the overall health and survival of the Atlantic salmon population.

[0115] Moreover, monitoring the respiratory rate is useful for determining the optimal time for harvesting the Atlantic salmon. As the fish grow and approach market size, their respiratory rate may change, indicating physiological changes associated with maturation. By closely tracking these changes, aquaculture managers can make informed decisions on when to harvest the fish, ensuring that they are collected at the peak of their quality and market value.

[0116] In addition to its practical applications, monitoring the respiratory rate of freely swimming Atlantic salmon also contributes to the scientific understanding of their biology and behavior. By collecting data on respiratory patterns under various environmental conditions and stressors, researchers can gain valuable insights into the adaptive responses and resilience of Atlantic salmon in aquaculture settings. This knowledge can guide the development of improved farming practices, breeding programs, and conservation efforts for this important species.

[0117] The fish farming enclosure 104 is an underwater structure designed to house and raise fish in an aquaculture environment 100. This enclosure 104 provides a controlled space where the fish 102 can swim freely while being monitored and cared for by the aquaculture operators. The specific design and characteristics of the fish farming enclosure 104 can vary depending on the species being raised, the location of the farm, and the farming methods employed.

[0118] In the case of farming Atlantic salmon, several types of fish farming enclosures are commonly used. One popular option is the open net pen system. These pens are typicallyconstructed using a floating framework made of high-density polyethylene (HDPE) pipes, which support a large, mesh net that extends underwater. The net pen is anchored to the seabed and can be located in sheltered coastal areas, fjords, or offshore waters. The mesh size of the net is carefully selected to allow water to flow freely through the pen while preventing the salmon from escaping and keeping predators out. Open net pens provide a spacious environment for the salmon to swim and interact, mimicking their natural habitat.

[0119] Another type of fish farming enclosure suitable for Atlantic salmon is the closed containment system. These systems aim to minimize the interaction between the farmed fish and the surrounding environment by creating a more controlled and isolated rearing space. Closed containment systems can be land-based or floating, and they typically involve the use of tanks, raceways, or large enclosed pens. Water is pumped into the system, circulated, and treated to maintain optimal conditions for the salmon. Closed containment systems offer greater control over water quality, temperature, and disease management, reducing the potential environmental impact and escaped fish compared to open net pens.

[0120] A third example of a fish farming enclosure for Atlantic salmon is the submerged cage system. These cages are like open net pens but are designed to be fully submerged underwater, typically at depths of 10 to 50 meters. Submerged cages offer several advantages, such as reduced exposure to surface weather conditions, improved water quality, and lower risk of fish escapes. The cages are constructed using strong, flexible materials like Kevlar or Dyneema and are anchored to the seabed using mooring lines. Submerged cages provide a more stable and controlled environment for the salmon, promoting their growth and well-being.

[0121] The underwater camera 106 is responsible for capturing video footage of the freely swimming fish 102, which forms the basis for analyzing their respiratory patterns and behavior. The camera 106 is strategically placed within the enclosure to ensure optimal coverage and visibility of the fish population.

[0122] The underwater camera 106 can be either stationary or mobile, depending on the specific requirements and layout of the fish farming enclosure 104. A stationary camera is fixed in a specific position within the enclosure 104, providing a constant and stable view of the fish 102 population. This setup is suitable for monitoring a specific area of interest or when the fish 102 are evenly distributed throughout the enclosure.

[0123] However, in some cases, a mobile camera system may be more advantageous for capturing comprehensive and dynamic footage of the freely swimming fish 102. A mobile camera allows for greater flexibility in monitoring different sections of the enclosure 104 and can adapt to changes in fish behavior or distribution.

[0124] One way to implement a mobile camera system is through manual control, such as a camera winch system. In this setup, the camera 106 is attached to a cable or winch that can be manually operated by the aquaculture staff. By adjusting the winch, the camera 106 can be raised, lowered, or moved horizontally within the enclosure. This manual control enables targeted monitoring of specific areas or individual fish, allowing for closer inspection when necessary.

[0125] In addition to, or as an alternative to, manual control, the mobile camera system can incorporate autonomous repositioning capabilities based on automated computer vision analysis. In this advanced setup, the camera 106 is equipped with intelligent software that continuously analyzes the video feed in real-time. By applying computer vision algorithms, the system can detect and track the movement of the fish 102, assess the quality of the captured footage, and determine the optimal position for the camera 106. An example of a computer vision and reinforcement learning-based technique for autonomously repositioning an underwater camera in an aquaculture environment is described in United States patent application no. 18 / 381,124, filed October 17, 2023, and entitled “POSITION POLICY DETERMINATION FOR UNDERWATER CAMERA POSITIONING IN AN AQUACULTURE ENVIRONMENT,” the entire contents of which is hereby incorporated by reference as if fully set forth herein.

[0126] For example, if the computer vision analysis detects that the fish 102 are congregating in a particular area of the enclosure, the autonomous system can automatically reposition the camera 106 to focus on that area, ensuring that high-quality video footage is captured. The system can also adjust the camera 106's position to maintain a clear view of the fish, even if they are moving rapidly or if the water conditions change.

[0127] Furthermore, the autonomous repositioning system can optimize the camera 106's position based on predefined criteria, such as maintaining a certain distance from the fish 102, capturing a specific number of individuals in the frame, or ensuring adequate lighting conditions. By continuously analyzing the video feed and making real-time adjustments, the mobile camera system can adapt to the dynamic nature of the aquaculture environment.

[0128] The autonomous repositioning capability of the mobile camera system offers several benefits. It reduces the need for constant manual intervention, allowing the aquaculture staff to focus on other tasks. It also ensures consistent and high-quality video footage, as the camera 106 can automatically adjust its position to maintain optimal viewing conditions. Additionally, the automated system can quickly respond to changes in fish behavior or environmental factors, providing a more comprehensive and accurate monitoring of the fish population.

[0129] The underwater camera 106 can be either monoscopic or stereoscopic, depending on the specific requirements and constraints of the monitoring system. A monoscopic camera, alsoknown as a single-view camera, captures a two-dimensional (2D) view of the underwater scene. It provides a straightforward and cost-effective solution for monitoring the fish, as it requires only one lens and image sensor. Monoscopic cameras are suitable for general observation and tracking of fish movement within the enclosure.

[0130] On the other hand, a stereoscopic camera system consists of two or more synchronized cameras that capture images from slightly different angles. By combining the information from multiple views, a stereoscopic camera can generate a three-dimensional (3D) representation of the underwater environment. This 3D capability enables more accurate tracking of individual fish, estimation of their sizes, and analysis of their spatial relationships. Stereoscopic cameras provide a richer dataset for the automated model to process, potentially enhancing the accuracy and reliability of the respiratory rate monitoring system.

[0131] In an embodiment, the underwater camera operates at a video frame rate of 8 frames per second (fps). This means that the camera captures and records 8 distinct images or frames every second, resulting in a smooth and continuous video stream. The choice of an 8 fps frame rate strikes a balance between temporal resolution and data storage requirements, but could be higher or lower according to the requirements of the particular implementation at hand.

[0132] A frame rate of 8 fps is considered sufficient for capturing the respiratory movements of the fish 102, as the typical breathing rates of Atlantic salmon fall within a range of 1 to 4 breaths per second. By capturing 8 frames per second, the camera ensures that each respiratory cycle is adequately sampled, allowing the automated model 108 to detect and analyze the mouth movements associated with breathing.

[0133] Moreover, operating at 8 fps helps to manage the volume of video data generated by the monitoring system. Higher frame rates would produce a larger amount of data, requiring increased storage capacity and processing power. By using an 8 fps frame rate, the system can efficiently handle the video stream while still providing sufficient temporal resolution for accurate respiratory rate analysis.

[0134] It is important to note that the specific frame rate used in the embodiment is not limited to 8 fps and can be adjusted based on the unique requirements of the aquaculture environment and the fish species being monitored. Higher frame rates may be necessary for capturing more rapid respiratory movements or for analyzing other behavioral aspects of the fish.

[0135] In addition to the frame rate, the underwater camera 106 can possess other characteristics to ensure reliable performance in a challenging aquaculture environment. The camera should be waterproof and resistant to corrosion, as it will be constantly exposed to saltwater or freshwater conditions. It should also have sufficient resolution and low-light sensitivity to capture clear and detailed images of the fish, even in dimly lit or turbid water conditions.

[0136] The automated model 108 utilizes computer vision techniques and machine learning algorithms to analyze the video footage captured by the underwater camera in the fish farming enclosure. A purpose of the automated model 108 is to monitor the respiratory patterns and health status of the freely swimming fish 102, such as Atlantic salmon, without the need for manual intervention.

[0137] The automated model 108 operates by first applying computer vision techniques to the video frames captured by the camera 106. These techniques enable the model to detect, track, and analyze the movement of individual fish within the enclosure 104. The model 108 uses advanced algorithms to identify and isolate each fish, creating a unique track for every individual. Each track consists of a subset of time-ordered video frames that follow the specific fish across the duration of the video.

[0138] Once the tracks are established, the automated model 108 employs a trained machine learning classifier 110 to determine the respiratory states of the fish. The classifier 110 analyzes each video frame within a track and identifies the mouth state of the fish, categorizing it as either open or closed. By examining the sequence of mouth states over time, the model 108 can determine the breathing pattern of each individual fish.

[0139] The machine learning classifier 110 used in the automated model 108 is pre-trained on a large dataset of labeled fish images, where the mouth states have been annotated. Through this training process, the classifier 110 learns to recognize the visual cues and features associated with open and closed mouth states. The trained classifier 110 can then be applied to new video footage to automatically determine the mouth states of a sample of the fish 102 without the need for manual labeling.

[0140] After analyzing the mouth states for each track, the automated model 108 aggregates the results to calculate the overall mouth state transition count. This count represents the total number of times the fish 102 in the enclosure 104 captured in the video transition between open and closed mouth states. By considering the collective breathing patterns of a sample of the fish 102 captured in the video, the model 108 can provide a comprehensive assessment of the respiratory activity within the population.

[0141] Using the mouth state transition count and the duration of the video footage, the automated model 108 calculates the average breathing rate of the fish 102. This metric provides a quantitative measure of the respiratory health and stress levels of the fish population. The average breathing rate can be compared to established baselines or thresholds to identify any deviations from normal behavior, indicating potential health issues or environmental stressors.

[0142] Finally, the automated model 108 outputs the calculated average breathing rate to a user- friendly interface, such as a graphical user interface (GUI) 112, a database 114, or a report 116.This output allows the aquaculture operators to easily monitor the respiratory health of the fish population and make informed decisions regarding their management and care.

[0143] The automated model 108 offers several advantages over manual monitoring methods. It enables continuous and real-time analysis of the fish population, providing timely insights into their health status. The model 108 can process large volumes of video data efficiently, reducing the workload on human observers and minimizing the risk of human error. Additionally, the machine learning approach allows the model 108 to adapt and improve over time as it learns from new data, enhancing its accuracy and reliability.

[0144] The automated model 108 can be integrated into the aquaculture environment 100, specifically within the fish farming enclosure 104 or its vicinity. This integration allows for efficient data processing and real-time monitoring of the fish population. One approach is to embed the automated model as a component of the underwater camera system itself.

[0145] In this setup, the camera 106 is equipped with advanced processing capabilities, such as an integrated microprocessor or a small single-board computer. The automated model 108's software, including the computer vision algorithms and machine learning classifier, is installed directly onto the camera 106's processing unit. By having the model reside within the camera 106, the video footage can be analyzed in real-time, eliminating the need for raw video data to be transmitted to a remote location for processing.

[0146] Alternatively, the automated model 108 can be implemented as a component of one or more programmable electronic devices located at, on, or near the fish farming enclosure 104. These devices can include dedicated computer systems, such as industrial PCs or embedded systems, specifically designed for aquaculture monitoring applications. The electronic devices are equipped with sufficient processing power, memory, and storage to execute the automated model 108's software and store the necessary data.

[0147] By placing the programmable electronic devices in close proximity to the fish farming enclosure 104, the system can minimize latency and ensure rapid data transfer between the camera 106 and the automated model 108. This proximity enables real-time analysis of the video footage and allows for immediate feedback and alerts to the aquaculture operators if any abnormalities or health concerns are detected.

[0148] The specific configuration of the programmable electronic devices can vary depending on the scale and complexity of the aquaculture setup. In smaller operations, a single device may suffice to handle the processing requirements of the automated model 108. However, in larger or more complex environments, a distributed network of devices may be employed to ensure optimal performance and reliability.

[0149] Having the automated model 108 located within the aquaculture environment 100 offers several advantages. It reduces the reliance on external infrastructure, such as cloud computing services or remote data centers, which can be subject to network disruptions or data transfer limitations. By processing the data locally, the system can provide faster and more reliable results, enabling timely decision-making and interventions by the aquaculture staff.

[0150] Moreover, the integration of the automated model 108 within the aquaculture environment 100 allows for a more self-contained and autonomous monitoring system. The model 108 can continuously analyze the video footage without the need for constant human supervision or manual data transfer. This autonomy reduces the workload on the aquaculture staff and allows them to focus on other critical aspects of fish farming operations.

[0151] In summary, the automated model 108 can be strategically located within the aquaculture environment 100, either as a component of the underwater camera system or as part of one or more programmable electronic devices positioned at, on, or near the fish farming enclosure. This integration enables real-time data processing, minimizes latency, and provides a more autonomous and efficient monitoring system for assessing the respiratory health and well-being of the freely swimming fish population.

[0152] Just as the automated model 108 can be integrated into the aquaculture environment 100, the database 114, graphical user interface (GUI) 112, or report 116 to which the average breathing rate is output can also be located within the same vicinity, such as on, at, or near the fish farming enclosure 104. This localized setup ensures that the critical information about the fish population's respiratory health is readily accessible to the aquaculture staff, enabling them to make timely decisions and take necessary actions.

[0153] When the database 114 is located within the aquaculture environment, it serves as a centralized repository for storing the average breathing rate data generated by the automated model 108. The database 114 can be hosted on a local server or a dedicated storage device, such as a network-attached storage (NAS) system, placed in close proximity to the fish farming enclosure 104. By storing the data locally, the system minimizes the need for remote data transmission, reducing potential delays or connectivity issues.

[0154] Similarly, the GUI 112 can be implemented on a local computing device, such as a desktop computer, laptop, or tablet, situated near the fish farming enclosure 104. The GUI 112 provides a user-friendly interface for the aquaculture staff to interact with the automated model 108 and view the real-time average breathing rate data. By having the GUI 112 accessible onsite, the staff can quickly monitor the respiratory health of the fish population and identify any abnormalities or trends that require attention.

[0155] In cases where the average breathing rate is output to a report 116, the report generation process can also be executed locally within the aquaculture environment 100. The automated model 108 can compile the relevant data and generate the report on a regular basis, such as daily or weekly, depending on the specific monitoring requirements. The report 116 can be stored on the local database 114 or distributed to the aquaculture staff via email or a local network shared folder.

[0156] Having the database, GUI, or report generation capabilities located within the aquaculture environment 100 offers several benefits. It enables the aquaculture staff to have immediate access to the critical information they need to make informed decisions about fish health management. They can quickly review the average breathing rate data, assess any deviations from normal levels, and take proactive measures to address potential issues.

[0157] Moreover, the localized setup reduces the reliance on external infrastructure and internet connectivity. In remote or offshore aquaculture facilities, internet access may be limited or unreliable. By keeping the data and user interfaces within the local environment, the monitoring system can continue to function effectively even in the absence of a stable internet connection.

[0158] The localized approach also enhances data security and privacy. By storing the average breathing rate data within the aquaculture facility, the system minimizes the risk of unauthorized access or data breaches that may occur during remote data transmission. The data remains under the control and supervision of the aquaculture staff, ensuring the confidentiality and integrity of the information.

[0159] While it can be advantageous to have some or all of the automated model 108, database 114, GUI 112, and report 116 located within the aquaculture environment 100, it is important to note that these components can also be remotely located, such as in a cloud or data center facility. This remote setup offers flexibility and scalability to the aquaculture monitoring system, allowing for centralized data processing, storage, and access.

[0160] In a remote configuration, the automated model 108 can be deployed on cloud-based servers or in a data center facility. These remote computing resources offer high-performance processing capabilities and can handle the computationally intensive tasks of analyzing the video footage and running the machine learning algorithms. By leveraging the power of cloud computing or dedicated data center infrastructure, the automated model 108 can process large volumes of data efficiently and scale according to the needs of the aquaculture operation.

[0161] Similarly, the database 114 can be hosted on remote servers or cloud platforms, providing a centralized repository for storing the average breathing rate data and other relevant information. Remote databases offer several advantages, such as increased storage capacity, data redundancy, and automatic backup mechanisms. By storing the data remotely, the aquaculturefacility can ensure the long-term preservation and accessibility of the information, even if the local infrastructure experiences issues or failures.

[0162] The GUI 112 can also be accessed remotely through web-based interfaces or remote desktop connections. This allows the aquaculture staff to monitor the fish population's respiratory health from anywhere, using any device with internet access. Remote access to the GUI 112 provides flexibility and convenience, enabling the staff to keep track of the average breathing rate and other critical metrics even when they are not physically present at the fish farming enclosure.

[0163] In terms of reporting, the generated reports 116 can be stored on remote servers or cloud storage services, making them easily accessible to authorized personnel. Remote storage of reports facilitates collaboration and information sharing among the aquaculture staff, researchers, and other stakeholders. It also enables the creation of centralized repositories for historical data analysis and long-term trend monitoring.

[0164] The remote location of some or all of these components offers several benefits. It allows for the centralization of data processing and storage, making it easier to manage and maintain the monitoring system. Remote servers and cloud platforms typically provide robust security measures, regular software updates, and data backup mechanisms, ensuring the integrity and availability of the data.

[0165] Moreover, remote access to the automated model 108, database 114, GUI 112, or reports 116 enables remote monitoring and decision-making. Aquaculture experts and researchers can access the data and insights from anywhere in the world, fostering collaboration and knowledge sharing. This remote accessibility is particularly valuable for large-scale aquaculture operations with multiple farming sites or for organizations with geographically dispersed teams.

[0166] In the example depicted in Figure 1, the automated model 108 is wirelessly connected to the camera 106 via a wireless access point affixed to the enclosure. The camera 106 is physically connected to the access point through a wired connection.

[0167] In this setup, the automated model 108 receives video data captured by the camera 106 through the wireless connection. This wireless communication allows the automated model 108 to be located separately from the camera 106 and the enclosure 104, providing flexibility in terms of placement and accessibility.

[0168] In some embodiments, the camera 106 generates crops of the video frames it captures. These crops specifically focus on the detected fish within the frames. To achieve this, the camera may employ a machine learning model, such as a Convolutional Neural Network (CNN), which is trained for object detection and tracking.

[0169] The CNN model analyzes each video frame captured by the camera and identifies the presence and location of fish within the image. It then generates crops around the detected fish, isolating them from the background. By sending only these fish-specific crops to the automated model 108, rather than the entire video frame, the camera 106 can optimize bandwidth usage and improve the efficiency of the wireless data transmission.

[0170] Once the automated model 108 receives the fish crops from the camera 106, it processes them using a trained classifier 110 to determine the respiratory states of the fish. The automated model 108 tracks the fish, analyzes their mouth states, and calculates the average breathing rate.

[0171] The results of this analysis, such as the average breathing rate of 0.75 breaths per second shown in the figure, can be output to one or more various destinations. These include a graphical user interface (GUI) 112 for real-time monitoring, a database 114 for storage and historical analysis, and a report 116 for documentation and sharing of the findings.

[0172] To train a machine learning classifier 110 for detecting and tracking fish that swim in front of the camera lens over a period of time ranging, for example, from 0.125 seconds to multiple seconds, several approaches can be employed. In an embodiment, a Convolutional Neural Network (CNN) architecture is used.

[0173] The training process can involve collecting a large dataset of video frames capturing fish swimming in an aquaculture environment. These frames should be annotated with bounding boxes or segmentation masks indicating the location and boundaries of the fish in each frame. The dataset can cover a diverse range of fish sizes, orientations, and swimming patterns to ensure the classifier's robustness.

[0174] During training, the CNN model can learn to recognize the visual features and patterns associated with fish across multiple frames. For example, the model can be trained using a sliding window approach, where it processes short sequences of frames (e.g., 0.125 seconds to 2 seconds) to learn the temporal dynamics of fish movement.

[0175] The CNN architecture can include convolutional layers to extract relevant features from the input frames, followed by recurrent layers (e.g., LSTM or GRU) to capture the temporal dependencies between frames. The output of the model can be the predicted bounding boxes or segmentation masks for the fish in each frame.

[0176] To handle the detection and tracking of multiple fish simultaneously, the model can employ techniques such as region proposal networks (RPN) or anchor boxes. These techniques allow the model to generate multiple object proposals and track them independently across frames.

[0177] During training, the model 110 can be optimized using appropriate loss functions, such as binary cross-entropy for object presence and mean squared error for bounding boxcoordinates. The model 110 can be trained iteratively, adjusting its parameters to minimize the difference between the predicted and ground truth annotations.

[0178] Data augmentation techniques, such as random cropping, flipping, and scaling, can be applied to enhance the model 110's ability to generalize to different fish appearances and orientations. Additionally, techniques like transfer learning could be leveraged, where a pretrained model on a related task (e.g., general object detection) is fine-tuned on the fish detection dataset to accelerate training and improve performance.

[0179] Once trained, the machine learning classifier 110 is capable of detecting and tracking fish that swim in front of the camera lens over the specified time range. The classifier can process the incoming video stream in real-time, identifying the presence and location of fish in each frame and tracking their movement across consecutive frames.

[0180] The accuracy and reliability of the classifier can depend on factors such as the quality and diversity of the training data, the complexity of the CNN architecture, and the hyperparameters used during training. Continuous monitoring and evaluation of the classifier 110's performance in real -world scenarios might be necessary to ensure its effectiveness and make any required adjustments or updates.

[0181] Turning now to FIG. 2, each example track Track-1, Track-2, Track-3, and Track 4 represents a sequence of video frames capturing a fish swimming in front of the camera lens. The same fish or different fishes can be tracked by the tracks. However, each track preferably represents one tracked fish. The machine learning classifier 110 analyzes each frame and determines the mouth status of the tracked fish, classifying it as either open (O), uncertain / unknown / occluded (U), or closed (C).

[0182] In the first track, Track-1, the sequence "OOOUUCCC" indicates that the classifier 110 detected the fish's mouth as open (O) in the first three frames, followed by two frames where the mouth status was uncertain or occluded (U), and finally, three frames where the mouth was classified as closed (C). This track provides a clear example of a fish transitioning from an open mouth state to a closed mouth state, with a period of uncertainty in between.

[0183] The second track, Track-2, represents a shorter sequence of frames, "UUUO". In this case, the classifier 110 initially identified the mouth status as uncertain or occluded (U) for the first three frames, followed by a single frame where the mouth was classified as open (O). This track highlights a scenario where the fish's mouth status is mostly uncertain, with a brief instance of an open mouth detection.

[0184] In the third track, Track-3, the sequence ''OCCUUUUUUUUUUCOO'' depicts a more complex scenario. The classifier 110 detected the fish's mouth as open (O) in the first frame, followed by two frames where the mouth was classified as closed (C). Subsequently, there is along sequence of frames (10 frames) where the mouth status was uncertain or occluded (U). Finally, the track ends with a closed mouth detection (C) and two frames of an open mouth (O). This track illustrates a fish exhibiting a mix of mouth states, with a significant portion of the frames being classified as uncertain.

[0185] The fourth track, Track-4, is the shortest sequence, consisting of only two frames, "CC". In both frames, the classifier 110 determined the fish's mouth status as closed (C). This track represents a scenario where the fish's mouth remains closed throughout the observed period.

[0186] In an embodiment, aggregating the mouth state classifications from multiple tracks like these is done to determine the overall mouth state transition count. By analyzing the transitions between open (O) and closed (C) states across all the tracks, the automated model 10 can calculate the average breathing rate of the fish population.

[0187] In the given example, Track- 1 and Track-3 contain a total of three clear transitions between open and closed mouth states, which would contribute to the overall mouth state transition count. In contrast, Track-2 and Track-4 do not show complete transitions.

[0188] By considering a large number of such tracks, the automated model 108 can build a comprehensive understanding of the respiratory patterns and behavior of the fish in the aquaculture environment. The inclusion of the uncertain / unknown / occluded (U) classification helps account for frames where the mouth status is ambiguous or not clearly visible, ensuring a more accurate representation of the fish's respiratory activity.

[0189] Turning now to FIG. 3, in an embodiment, the system employs a strategy to handle tracks that contain an extended sequence of consecutive mouth status classifications as uncertain / unknown / occluded (U). This scenario is illustrated in the example Track-5, where a significant number of consecutive U classifications separate two clear transitions between open (O) and closed (C) mouth states.

[0190] To address this situation and accurately account for the transitions, the system introduces a threshold based on the number of consecutive U classifications. If the number of consecutive U classifications exceeds this threshold, the track is split into two separate tracks.

[0191] The first track comprises the sequence of mouth status classifications before the consecutive U sequence, while the second track contains the sequence of classifications after the consecutive U sequence.

[0192] For instance, in the provided example Track-5, if the threshold is set to the number of video frames corresponding to one second of time (or any other suitable time length) where each video frame corresponds to 0.125 seconds of time, the system might split the track into two tracks:

[0193] Track-5A: OCC

[0194] Track-5B: OOC

[0195] In this split, the first track (Track-5A) captures the initial transition from open to closed (OC), while the second track (Track-5B) includes the second clear transition from open to closed (OC) at the end of the sequence.

[0196] By splitting the track in this manner, the system avoids incorrectly counting the sequence of consecutive U classifications as a transition. Instead, it focuses on the clear transitions between open and closed mouth states, which provide more reliable information about the fish's respiratory behavior.

[0197] This approach has several advantages. By separating the track into two distinct tracks, the system can accurately identify and count the clear transitions between open and closed mouth states, avoiding potential errors introduced by extended periods of uncertainty or occlusion.

[0198] The threshold for splitting the track can be adjusted based on the specific requirements of the aquaculture environment 100 or the characteristics of the fish species being monitored. For example, if the fish exhibit longer periods of occlusion or uncertainty, the threshold can be increased to accommodate these scenarios.

[0199] The system becomes more resilient to challenges such as occlusions, poor visibility, or ambiguous mouth states. By focusing on the clear transitions and ignoring extended periods of uncertainty, the system can still provide reliable estimates of the average breathing rate, even in less-than-ideal conditions.

[0200] It is important to note that the choice of the threshold value should be carefully considered and may require calibration based on empirical observations and expert knowledge. Additionally, the system may incorporate other rules or heuristics to handle specific scenarios or edge cases that may arise during the analysis of the tracks.

[0201] Returning to FIG. 2, in this example, we have three clear transitions over the four tracks.The average breathing rate can be calculated according to the following equation: OPENING TRANSITIONS + CLOSING TRANSITIONS)

[0202] Here, OPENING TRANSITIONS represents the number of clear transitions from mouth closed to mouth open counted over all tracks. In the example of FIG. 2,OPENING TRANSITIONS is one (1). CLOSING TRANSITIONS represents the number of clear transitions from mouth open to mouth closed counted over all tracks. In the example of FIG. 2, CLOSING TRANSITIONS is two (2). TOTAL VIDEO TIME represents the time (e.g., in seconds) over which the transitions were observed. For example, assuming a frame rate of eight (8) frames per second and three (3) transitions observed over sixteen (16) total usablevideo frames (eight (8) from Track-1, zero (0) from Track-2, six (6) from Track-3, and two (2) from Track-4), then the average breathing rate is 0.75 breaths per second. Here, the TOTAL VIDEO TIME is calculated as 2 seconds (sixteen (16) usable video frames divided by the eight (8) of frames per second at which video is captured).

[0203] METHOD FOR ESTIMATING FISH RESPIRATION ACTIVITY IN AN AQUACULTURE ENVIRONMENT

[0204] FIG. 4 is a flowchart of a method 400 for estimating fish respiration activity in an aquaculture environment. The method 400 outlines an automated system for monitoring and analyzing the breathing rate of freely swimming fish in an aquaculture environment. The system utilizes computer vision techniques and machine learning algorithms to process video footage captured by an underwater camera placed in a fish farming enclosure.

[0205] The method 400 begins at step 405 with the automated model using computer vision techniques to identify and track individual fish across a set of video frames. Each track represents a specific fish and consists of a subset of time-ordered video frames from the overall set of captured frames. This tracking process allows the system to focus on the behavior and characteristics of individual fish over time.

[0206] For each track, the automated model employs at step 410 a trained machine learning classifier to analyze the respective subset of time-ordered video frames, or fish crops thereof. The classifier is specifically designed to determine the mouth state of the fish in each frame or crop, classifying it as either open or closed. By examining the sequence of mouth states over time, the model can establish a set of time-ordered fish mouth states for each track.

[0207] Once the mouth states have been determined for all the tracks, the automated model at step 415 aggregates the information to calculate the overall mouth state transition count. This count represents the total number of times the fish in the enclosure transition between open and closed mouth states. The aggregation process takes into account the data from all the tracked fish, providing a comprehensive view of the respiratory activity within the population.

[0208] Using the overall mouth state transition count, the automated model then calculates at 420 the average breathing rate of the fish. This calculation involves dividing the total number of mouth state transitions by the duration of the analyzed video footage, resulting in an estimate of the number of breaths taken by the fish per unit of time.

[0209] Finally, the automated model at step 425 outputs the determined average breathing rate to one or more destinations, such as a graphical user interface (GUI), a database, or a report. The GUI allows for real-time monitoring and visualization of the breathing rate, enabling aquaculture operators to quickly assess the respiratory health of the fish population. The database serves as a repository for storing historical breathing rate data, facilitating long-termanalysis and trend identification. The report provides a formal record of the findings, which can be shared with stakeholders or used for documentation purposes.

[0210] The method 400 offers several advantages over manual observation and analysis techniques. By automating the process of tracking fish, determining mouth states, and calculating the average breathing rate, the system reduces the reliance on human observers and minimizes the potential for errors or biases. The use of computer vision and machine learning algorithms enables the processing of large volumes of video data efficiently and consistently, allowing for continuous monitoring of the fish population.

[0211] Moreover, the system's ability to track individual fish and aggregate data from multiple tracks provides a more comprehensive and accurate assessment of the overall respiratory health of the fish population. This information can be invaluable for aquaculture managers in making informed decisions regarding fish welfare, environmental conditions, and feeding strategies.

[0212] In an embodiment, the method 400 can be adapted to detect ram ventilation, a respiratory behavior exhibited by some fish species where the mouth remains constantly open. In ram ventilation, fish swim forward with their mouth and operculum (gill cover) continuously open, allowing a constant flow of water over their gills for oxygen exchange.

[0213] To adapt the method 400 for detecting ram ventilation, the trained machine learning classifier could be modified to recognize and classify the mouth state as either "open" or "ram ventilating." In addition to or instead of focusing on the transition between open and closed mouth states, the classifier would be trained to identify instances where the fish's mouth remains open for an extended period, indicating ram ventilation.

[0214] The training process for the classifier could involve providing labeled examples of video frames showcasing fish exhibiting ram ventilation behavior. These examples would highlight the key characteristics of ram ventilation, such as the consistently open mouth and the fish's forward swimming motion. By learning from these examples, the classifier would develop the ability to distinguish between normal open mouth states and the prolonged open mouth state associated with ram ventilation.

[0215] Once the classifier is trained, the automated model would process the video frames. However, in addition to or instead of counting the transitions between open and closed mouth states, the model would also keep track of the duration of consecutive "open" or "ram ventilating" mouth states for each fish track.

[0216] If a fish track exhibits a prolonged sequence of "open" or "ram ventilating" mouth states, exceeding a predefined threshold duration, the model would identify that fish as engaging in ram ventilation. This threshold duration could be determined based on the specific characteristics of the fish species and their typical respiratory behavior.

[0217] The model would then calculate the proportion of fish exhibiting ram ventilation behavior within the analyzed population. This information could be presented alongside the average breathing rate, providing a more comprehensive assessment of the respiratory health and behavior of the fish population.

[0218] In the adapted method, the output of the automated model could include both the average breathing rate and the prevalence of ram ventilation behavior. The graphical user interface (GUI), database, and report would be modified to accommodate this additional information, allowing aquaculture operators to monitor and track the occurrence of ram ventilation within the fish population.

[0219] The detection of ram ventilation can provide valuable insights into the health and wellbeing of the fish. Ram ventilation is often associated with increased stress or higher metabolic demands, such as during periods of intense swimming or in response to environmental challenges. By identifying instances of ram ventilation, aquaculture managers can take appropriate measures to address potential stressors and optimize the conditions within the fish farming enclosure.

[0220] Moreover, understanding the prevalence of ram ventilation within the population can inform management decisions related to stocking densities, water quality, and feeding strategies. If a significant proportion of fish are exhibiting ram ventilation, it may indicate a need for interventions to improve the overall health and welfare of the fish population.

[0221] In an embodiment, the method 400 of Fig. 4 is extended by specifying that each track within the set of tracks includes additional tracking information. This tracking information indicates the respective location of the tracked fish in each video frame of the track's subset of time-ordered video frames. By incorporating this location data, the method can provide more precise information about the position and movement of individual fish within the fish farming enclosure. This information can be useful for generating fish crops and for analyzing fish behavior, monitoring their spatial distribution, and detecting any anomalies or patterns that may be relevant to their health and well-being.

[0222] In an embodiment, the method 400 of Fig. 4 is enhanced by introducing the concept of bounding box information for each track. The bounding box information specifies the respective area within each video frame where at least a portion of the tracked fish is displayed. This means that instead of just tracking the fish's location as a single point, the method now considers a rectangular region that encompasses the fish's body. By using bounding boxes, the method 400 can capture more detailed information about the fish's size, orientation, and any visible physical characteristics. This additional data can be valuable for generating fish crops and for assessing the fish's growth, detecting any signs of injury or disease, and monitoring their overall condition.

[0223] In an embodiment, the trained machine learning classifier used in the method 400 is specifically designed to classify the mouth state of a fish displayed in an image into three categories: open, closed, or unknown. This categorization allows the method 400 to handle situations where the fish's mouth state is not clearly visible or discernible. By including an "unknown" category, the classifier can account for instances where the mouth is obscured, partially visible, or in a transitional state between open and closed. This additional classification helps to improve the accuracy and reliability of the breathing rate estimation by acknowledging the presence of uncertainty in some video frames.

[0224] In an embodiment, a specific approach for determining the overall mouth state transition count involves calculating two separate counts: the first count represents the number of transitions from an open mouth state to a closed mouth state, while the second count represents the number of transitions from a closed mouth state to an open mouth state. These counts are derived from the sets of time-ordered fish mouth states obtained for each track. To determine the overall mouth state transition count, the method simply adds the first count and the second count together. This approach provides a more detailed breakdown of the mouth state transitions, allowing for a more comprehensive analysis of the fish's breathing patterns.

[0225] In an embodiment, a scenario is addressed where a particular set of time-ordered fish mouth states contains a subset of consecutive "unknown" states. If the duration of this subset of unknown states exceeds a predefined threshold amount of time, the method 400 determines that the particular set of mouth states should be split into multiple sets. This splitting process is based on the assumption that a prolonged period of unknown mouth states may indicate a significant change in the fish's behavior or environment. By dividing the original set of mouth states into multiple subsets, the method can separately analyze the breathing patterns before and after the extended period of unknown states. This approach helps to maintain the accuracy and reliability of the overall mouth state transition count by accounting for potential discontinuities or anomalies in the fish's behavior.

[0226] In an embodiment, a minimum requirement for the number of video frames in each track's subset of time-ordered video frames is specified. According to this embodiment, each subset must contain at least three video frames. This requirement ensures that the method has sufficient data points to analyze the fish's mouth state transitions accurately. By having at least three video frames, the method can capture at least one complete cycle of mouth opening and closing, providing a more reliable basis for estimating the breathing rate. This embodiment helps to establish a baseline for the temporal resolution and data quality needed for effective breathing rate monitoring.

[0227] EXAMPLE PROGRAMMABLE ELECTRONIC DEVICE

[0228] FIG. 5 illustrates an example of a programmable electronic device 500 that processes and manipulates data to perform techniques disclosed herein for estimating fish respiratory activity in an aquaculture environment. For example, one or more programmable electronic devices 500 may be used to implement the automated model disclosed herein. Example programmable electronic device 500 includes electronic components encompassing hardware or hardware and software including processor 502, memory 504, auxiliary memory 506, input device 508, output device 510, mass data storage 512, and network interface 514, all connected to bus 516. Network 522 is connected to, but not part of, programmable electronic device 500.

[0229] While only one of each type of component is depicted in FIG. 5 for the purpose of providing a clear example, multiple instances of any or all these electronic components are present in device 500 in other instances. For example, in an embodiment, multiple processors (including, potentially, multiple different types of processors) are connected to bus 516. Accordingly, unless the context clearly indicates otherwise, reference with respect to FIG. 5 to a component of device 500 in the singular such as, for example, processor 502, is not intended to exclude the plural where, in a particular instance of device 500, multiple instances of the electronic component are present.

[0230] Processor 502 is an electronic component that processes (e.g., executes, interprets, or otherwise processes) instructions 518 including instructions 520 for estimating fish respiratory activity in an aquaculture environment. In an embodiment, processor 502 fetches, decodes, and executes instructions 518 from memory 504 and performs arithmetic and logic operations dictated by instructions 518 and coordinates the activities of other electronic components of device 500 in accordance with instructions 518. In an embodiment, processor 502 is made using silicon wafers according to a manufacturing process (e.g., 7nm, 5nm, or 3nm). In an embodiment, processor 502 is configured to understand and execute a set of commands referred to as an instruction set architecture (ISA) (e.g., x86, x86_64, or ARM).

[0231] In an embodiment, processor 502 includes a cache used to store frequently accessed instructions 518 to speed up processing. In an embodiment, processor 502 has multiple layers of cache (LI, L2, L3) with varying speeds and sizes.

[0232] In an embodiment, processor 502 is composed of multiple cores where each such core is a processor within processor 502. The cores allow processor 502 to process multiple instructions 518 at once in a parallel processing manner.

[0233] In an embodiment, processor 502 supports multi -threading where each core of processor 502 handles multiple threads (multiple sequences of instructions) at once to further enhance parallel processing capabilities.

[0234] In an embodiment, processor 502 is any of the following types of central processing units (CPUs): a desktop processor for general computing, gaming, content creation, etc.; a server processor for data centers, enterprise-level applications, cloud services, etc.; a mobile processor for portable computing devices like laptops and tablets for enhanced battery life and thermal management; a workstation processor for intense computational tasks like 3D rendering and simulations; or any other type of CPU suitable for the particular implementation at hand.

[0235] While processor 502 might be a CPU, processor 502, in an embodiment, is any of the following types of processors: a graphics processing unit (GPU) capable of highly parallel computation allowing for processing of multiple calculations simultaneously and useful for rendering images and videos and for accelerating machine learning computation tasks; a digital signal processor (DSP) designed to process analog signals like audio and video signals into digital form and vice versa, commonly used in audio processing, telecommunications, and digital imaging; specialized hardware for machine learning workloads, especially those involving tensors (multi-dimensional arrays); a field-programmable gate array (FPGA) or other reconfigurable integrated circuit that is customized post-manufacturing for specific applications, such as cryptography, data analytics, and network processing; a neural processing unit (NPU) or other dedicated hardware designed to accelerate neural network and machine learning computations, commonly found in mobile devices and edge computing applications; an image signal processor (ISP) specialized in processing images and videos captured by cameras, adjusting parameters like exposure, white balance, and focus for enhanced image quality; an accelerated processing unit (APU) combing a CPU and a GPU on a single chip to enhance performance and efficiency, especially in consumer electronics like laptops and consoles; a vision processing unit (VPU) dedicated to accelerating machine vision tasks such as image recognition and video processing, typically used in drones, cameras, and autonomous vehicles; a microcontroller unit (MCU) or other integrated processor designed to control electronic devices, containing CPU, memory, and input / output peripherals; an embedded processor for integration into other electronic devices such as washing machines, cars, industrial machines, etc.; a system on a chip (SoC) such as those commonly used in smartphones encompassing a CPU integrated with other components like a graphics processing unit (GPU) and memory on a single chip; or any other type of processor suitable for the particular implementation at hand.

[0236] Memory 504 is an electronic component that stores data and instructions 518 that processor 502 processes. In an embodiment, memory 504 provides the space for the operating system, applications, and data in current use to be quickly reached by processor 502. In an embodiment, memory 504 is a random-access memory (RAM) that allows data items to be reador written in substantially the same amount of time irrespective of the physical location of the data items inside memory 504.

[0237] In an embodiment, memory 504 is a volatile or non-volatile memory. Data stored in a volatile memory is lost when the power is turned off. Data in non-volatile memory remains intact even when the system is turned off. In an embodiment, memory 504 is Dynamic RAM (DRAM). DRAM such as Single Data Rate RAM (SDRAM) or Double Data Rate RAM (DDRAM) is volatile memory that stores each bit of data in a separate capacitor within an integrated circuit. The capacitors of DRAM leak charge and need to be periodically refreshed to avoid information loss. In an embodiment, memory 504 is Static RAM (SRAM). SRAM is volatile memory that is typically faster but more expensive than DRAM. SRAM uses multiple transistors for each memory cell but does not need to be periodically refreshed. Additionally, or alternatively, SRAM is used for cache memory in processor 502 in an embodiment. In an embodiment, memory 504 encompasses both DRAM and SRAM.

[0238] Device 500 has auxiliary memory 506 other than memory 504. Examples of auxiliary memory 506 include cache memory, register memory, read-only memory (ROM), secondary storage, virtual memory, memory controller, and graphics memory. In an embodiment, device 500 has multiple auxiliary memories including different types of auxiliary memories.

[0239] Cache memory is found inside or very close to processor 502 and is typically faster but smaller than memory 504. Cache memory is used to hold frequently accessed instructions 518 (encompassing any associated data) to speed up processing. In an embodiment, cache memory is hierarchical ranging from Level 1 cache memory which is the smallest but fastest cache memory and is typically inside processor 502 to Level 2 and Level 3 cache memory which are progressively larger and slower cache memories that are inside or outside processor 502.

[0240] Register memory is a small but very fast storage location within processor 502 designed to hold data temporarily for ongoing operations.

[0241] ROM is a non-volatile memory device that is only read, not written to. In an embodiment, ROM is a Programmable ROM (PROM), Erasable PROM (EPROM), or electrically erasable PROM (EEPROM). In an embodiment, ROM stores basic input / output system (BIOS) instructions which help device 500 boot up.

[0242] Secondary storage is a non-volatile memory. In an embodiment, secondary storage encompasses any or all of: a hard disk drive (HDD) or other magnetic disk drive device; a solid- state drive (SSD) or other NAND-based flash memory device; an optical drive like a CD-ROM drive, a DVD drive, or a Blu-ray drive; or flash memory device such as a USB drive, an SD card, or other flash storage device.

[0243] Virtual memory is a portion of a hard drive or an SSD that the operating system uses as if it were memory 504. When memory 504 gets filled, less frequently accessed data and instructions 518 is “swapped” out to the virtual memory. The virtual memory is slower than memory 504, but it provides the illusion of having a larger memory 504.

[0244] A memory controller manages the flow of data and instructions 518 to and from memory 504. The memory controller is located either on the motherboard of device 500 or within processor 502.

[0245] Graphics memory is used by a graphics processing unit (GPU) and is specially designed to handle the rendering of images, videos, graphics, or performing machine learning calculations. Examples of graphics memory include graphics double data rate (GDDR) such as GDDR5 and GDDR6.

[0246] Input device 508 is an electronic component that allows users to feed data and control signals into device 500. Input device 508 translates a user’s action or the data from the external world into a form that device 500 processes. Examples of input device 508 include a keyboard, a pointing device (e.g., a mouse), a touchpad, a touchscreen, a microphone, a scanner, a webcam, a joystick / game controller, a graphics tablet, a digital camera, a barcode reader, a biometric device, a sensor, and a MIDI instrument.

[0247] Output device 510 is an electronic component that conveys information from device 500 to the user or to another device. The information is in the form of text, graphics, audio, video, or other media representation. Examples of output device 510 include a monitor or display device, a printer device, a speaker device, a headphone device, a projector device, a plotter device, a braille display device, a haptic device, a LED or LCD panel device, a sound card, and a graphics or video card.

[0248] Mass data storage 512 is an electronic component used to store data and instructions 518. In an embodiment, mass data storage 512 is non-volatile memory. Examples of mass data storage 512 include a hard disk drive (HDD), a solid-state drive (SDD), an optical drive, a flash memory device, a magnetic tape drive, a floppy disk, an external drive, or a RAID array device.

[0249] In an embodiment, mass data storage 512 is additionally or alternatively connected to device 500 via network 522. In an embodiment, mass data storage 512 encompasses a network attached storage (NAS) device, a storage area network (SAN) device, a cloud storage device, or a centralized network filesystem device.

[0250] Network interface 514 (sometimes referred to as a network interface card, NIC, network adapter, or network interface controller) is an electronic component that connects device 500 to network 522. Network interface 514 functions to facilitate communication between device 500 and network 522. Examples of a network interface 514 include an ethemet adaptor, a wirelessnetwork adaptor, a fiber optic adapter, a token ring adaptor, a USB network adaptor, a Bluetooth adaptor, a modem, a cellular modem or adapter, a powerline adaptor, a coaxial network adaptor, an infrared (IR) adapter, an ISDN adaptor, a VPN adaptor, and a TAP / TUN adaptor.

[0251] Bus 516 is an electronic component that transfers data between other electronic components of or connected to device 500. Bus 516 serves as a shared highway of communication for data and instructions (e.g., instructions 518), providing a pathway for the exchange of information between components within device 500 or between device 500 and another device. Bus 516 connects the different parts of device 500 to each other. In an embodiment, bus 516 encompasses one or more of a system bus, a front-side bus, a data bus, an address bus, a control bus, an expansion bus, a universal serial bus (USB), a I / O bus, a memory bus, an internal bus, an external bus, and a network bus.

[0252] Instructions 518 are computer-processable instructions that take different forms. In an embodiment, instructions 518 are in a low-level form such as binary instructions, assembly language, or machine code according to an instruction set (e.g., x86, ARM, MIPS) that processor 502 is designed to process. In an embodiment, instructions 518 include individual operations that processor 502 is designed to perform such as arithmetic operations (e.g., add, subtract, multiply, divide, etc.); logical operations (e.g., AND, OR, NOT, XOR, etc.); data transfer operations including moving data from one location to another such as from memory 504 into a register of processor 502 or from a register to memory 504; control instructions such as jumps, branches, calls, and returns; comparison operations; and specialization operations such as handling interrupts, floating-point arithmetic, and vector and matrix operations. In an embodiment, instructions 518 are in a higher-level form such as programming language instructions in a high-level programming language such as Python, Java, C++, etc. In an embodiment, instructions 518 are in an intermediate level form in between a higher-level form and a low-level form such as bytecode or an abstract syntax tree (AST).

[0253] Instructions 518 for processing by processor 502 are in different forms at the same or different times. In an embodiment, when stored in mass data storage 512 or memory 504, instructions 518 are stored in a higher-level form such as Python, Java, or other high-level programing language instructions, in an intermediate-level form such as Python or Java bytecode that is compiled from the programming language instructions, or in a low-level form such as binary code or machine code. In an embodiment, when stored in processor 502, instructions 518 are stored in a low-level form such as binary instructions, assembly language, or machine code according to an instruction set architecture (ISA). In an embodiment, instructions 518 are stored in processor 502 in an intermediate level form or even a high-level form where CPU 502 processes instructions in such form.

[0254] Instructions 518 are processed by one or more processors of device 500 using a processing model such as any or all of the following processing models: sequential execution where instructions are processed one after another in a sequential manner; pipelining where pipelines are used to process multiple instruction phases concurrently; multiprocessing where different processors different instructions concurrently, sharing the workload; thread-level parallelism where multiple threads run in parallel across different processors; simultaneous multithreading or hyperthreading where a single processor processes multiple threads simultaneously, making it appear as multiple logical processors; multiple instruction issue where multiple instruction pipelines allow for the processing of several instructions during a single clock cycle; parallel data operations where a single instruction is used to perform operations on multiple data elements concurrently; clustered or distributed computing where multiple processors in a network (e.g., in the cloud) collaboratively process the instructions, distributing the workload across the network; graphics processing unit (GPU) acceleration where GPUs with their many processors allow the processing of numerous threads in parallel, suitable for tasks like graphics rendering and machine learning; asynchronous execution where processing of instructions is driven by events or interrupts, allowing the one or more processors to handle tasks asynchronously; concurrent instruction phases where multiple instruction phases (e.g., fetch, decode, execute) of different instructions are handled concurrently; parallel task processing where different processors handle different tasks or different parts of data, allowing for concurrent processing and execution; or any other processing model suitable to meet the requirements of the particular implementation at hand.

[0255] Network 522 is a collection of interconnected computers, servers, and other programmable electronic devices that allow for the sharing of resources and information. Network 522 ranges in size from just two connected devices to a global network (e.g., the internet) with many interconnected devices. In an embodiment, network 522 encompasses network devices such as routers, switches, hubs, modems, and access points.

[0256] Individual devices on network 522 are sometimes referred to as “network nodes.” Network nodes communicate with each other through mediums or channels sometimes referred to as “network communication links.” The network communication links are wired (e.g., twisted-pair cables, coaxial cables, or fiber-optic cables) or wireless (e.g., Wi-Fi, radio waves, or satellite links). Network nodes follow a set of rules sometimes referred to “network protocols” that define how the network nodes communicate with each other. Example network protocols include data link layer protocols such as Ethernet and Wi-Fi, network layer protocols such as IP (Internet Protocol), transport layer protocols such as TCP (Transmission Control Protocol), application layer protocols such as HTTP (Hypertext transfer Protocol) and HTTPS (HTTPSecure), and routing protocols such as OSPF (Open Shortest Path First) and BGP (Border Gateway Protocol).

[0257] Network 522 has a particular physical or logical layout or arrangement sometimes referred to as a “network topology.” Example network topologies include bus, star, ring, and mesh. In an embodiment, network 522 encompasses any or all of the following categories of networks: a personal area network (PAN) that covers a small area (a few meters), like a connection between a computer and a peripheral device via Bluetooth; a local area network (LAN) that covers a limited area, such as a home, office, or campus; a metropolitan area network (MAN) that covers a larger geographical area, like a city or a large campus; a wide area network (WAN) that spans large distances, often covering regions, countries, or even globally (e.g., the internet); a virtual private network (VPN) that provides a secure, encrypted network that allows remote devices to connect to a LAN over a WAN; an enterprise private network (EPN) build for an enterprise, connecting multiple branches or locations of a company; or a storage area network (SAN) that provides specialized, high-speed block-level network access to storage using high-speed network links like Fibre Channel.

[0258] TERMINOLOGY

[0259] As used herein and in the appended claims, the term “computer-readable media” refers to one or more mediums or devices that store or transmit information in a format that a computer system accesses. Computer-readable media encompasses both storage media and transmission media. Storage media includes volatile and non-volatile memory devices such as RAM devices, ROM devices, secondary storage devices, register memory devices, memory controller devices, graphics memory devices, and the like. Transmission media includes wired and wireless physical pathways that carry communication signals such as twisted pair cable, coaxial cable, fiber optic cable, radio waves, microwaves, infrared, visible light communication, and the like.

[0260] As used herein and in the appended claims, the term “non-transitory computer-readable media” encompasses computer-readable media as just defined but excludes transitory, propagating signals. Data stored on non-transitory computer-readable media isn’t just momentarily present and fleeting but has some degree of persistence. For example, instructions stored in a hard drive, a SSD, an optical disk, a flash drive, or other storage media are stored on non-transitory computer-readable media. Conversely, data carried by a transient electrical or electromagnetic signal or wave is not stored in non-transitory computer-readable media when so carried.

[0261] As used herein and in the appended claims, unless otherwise clear in context, the terms “comprising,” “having,” “containing,” “including,” “encompassing,” “in response to,” “based on,” and the like are intended to be open-ended in that an element or elements following such aterm is not meant to be an exhaustive listing of elements or meant to be limited to only the listed element or elements.

[0262] Unless otherwise clear in context, relational terms such as “first” and “second” are used herein and in the appended claims to differentiate one thing from another without limiting those things to a particular order or relationship. For example, unless otherwise clear in context, a “first device” could be termed a “second device.” The first and second devices are both devices, but not the same device.

[0263] Unless otherwise clear in context, the indefinite articles "a" and "an" are used herein and in the appended claims to mean “one or more” or “at least one.” For example, unless otherwise clear in context, “in an embodiment” means in at least one embodiment, but not necessarily more than one embodiment. Accordingly, unless otherwise clear in context, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices, unless otherwise clear in context, are collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B and C” encompasses both (a) a single processor configured to carry out recitations A, B, and C, (b) multiple processors each configured to carry out recitations A, B, and C, and (c) a first processor configured to carry out recitation A working in conjunction (as a team) with a second processor configured to carry out recitations B and C.

[0264] Unless otherwise clear in context, the terms “set,” and “collection” should generally be interpreted to include one or more described items throughout this application. Accordingly, unless otherwise clear in context, phrases such as “a set of one or more devices configured to” or “a collection of one or more devices configured to” are intended to include one or more recited devices. Such one or more recited devices, unless otherwise clear in context, are collectively configured to carry out the stated recitations. For example, “a set of one or more servers configured to carry out recitations A, B and C” encompasses both (a) a single server configured to carry out recitations A, B, and C, (b) multiple servers each configured to carry out recitations A, B, and C, and (c) a first server configured to carry out recitations A and B working in conjunction (as a team) with a second server configured to carry out recitation C.

[0265] As used herein, unless otherwise clear in context, the term “or” is open-ended and encompasses all possible combinations, except where infeasible. For example, if it is stated that a component includes A or B, then, unless infeasible or otherwise clear in context, the component includes at least A, or at least B, or at least A and B. As a second example, if it is stated that a component includes A, B, or C then, unless infeasible or otherwise clear in context, the component includes at least A, or at least B, or at least C, or at least A and B, or at least A and C, or at least B and C, or at least A and B and C.

[0266] Unless the context clearly indicates otherwise, conjunctive language in this description and in the appended claims such as the phrase "at least one of X, Y, and Z," is to be understood to convey that an item, term, etc. is either X, Y, or Z, or a combination thereof. Thus, such conjunctive language does not require that at least one of X, at least one of Y, and at least one of Z to each be present.

[0267] Unless the context clearly indicates otherwise, the relational term “based on” is used in this description and in the appended claims in an open-ended fashion to describe a logical (e.g., a condition precedent) or causal connection or association between two stated things where one of the things is the basis for or informs the other without requiring or foreclosing additional unstated things that affect the logical or causal connection or association between the two stated things.

[0268] Unless the context clearly indicates otherwise, the relational term “in response to” or “responsive to” is used in this description and in the appended claims in an open-ended fashion to describe a stated action or behavior that is done as a reaction or reply to a stated stimulus without requiring or foreclosing additional unstated stimuli that affect the relationship between the stated action or behavior and the stated stimulus.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: using a computer vision technique to determine a set of tracks from a set of video frames captured by a camera immersed underwater in an aquaculture environment comprising freely swimming fish; wherein each track of the set of tracks comprises a respective subset of time-ordered video frames, of the set of video frames, for the track; for each track of the set of tracks, using a trained machine learning classifier to determine, based on the respective subset of time-ordered video frames for the track, a respective set of time-ordered fish mouth states for the track; aggregating the respective sets of time-ordered fish mouth states determined for the set of tracks to determine an overall mouth state transition count; determining an average breathing rate based on the overall mouth state transition count; and outputting the average breathing rate to a graphical user interface, a database, or a report.

2. The method of claim 1, wherein each track of the set of tracks comprises respective tracking information indicating, for each video frame of the respective subset of time-ordered video frames, a respective location in the video frame of a fish that is tracked in the track.

3. The method of claim 1, wherein each track of the set of tracks comprises respective bounding box information indicating, for each video frame of the respective subset of time- ordered video frames, a respective area of the video frame that displays at least a portion of a fish that is tracked in the track.

4. The method of claim 1, wherein the trained machine learning classifier is trained to classify a fish mouth state displayed in an image as open, closed, or unknown.

5. The method of claim 1, further comprising: determining a first count of transitions from mouth open to mouth closed and a second count of transitions from mouth closed to mouth open from the sets of time- ordered fish mouth states; anddetermining the overall mouth state transition count as a sum of the first count and the second count.

6. The method of claim 1, wherein: a particular set of time-ordered fish mouth states of the respective sets of time-ordered fish mouth states comprises a subset of consecutive unknown fish mouth states; the method comprises determining that the subset of consecutive unknown fish mouth states corresponds to a time period that exceeds a threshold amount of time, and determining to split the particular set of fish mouth states into multiple sets of time-ordered fish mouth states based on the time period exceeding the threshold amount of time; and the aggregating the respective sets of time-ordered fish mouth states determined for the set of tracks to determine the overall mouth state transition count is based on the multiple sets of time-ordered fish mouth states.

7. The method of claim 1, wherein, for each track of the set of tracks, the respective subset of time-ordered video frames comprises at least three video frames.

8. The method of claim 1, further comprising: aggregating the respective sets of time-ordered fish mouth states determined for the set of tracks to determine an average frequency of mouth openings as an overall respiratory index; and outputting the overall respiratory index to a graphical user interface, a database, or a report.

9. The method of claim 1, wherein the machine learning classifier is trained to account for an oral valve when classifying a fish mouth displayed in an image as to fish mouth state.

10. The method of claim 1, further comprising: determining the average breathing rate based on a video frame rate of the camera.

11. The method of claim 1, wherein the machine learning classifier is trained to account for ram ventilation when classifying a fish mouth displayed in an image as to fish mouth state.

12. The method of claim 1, wherein each track of the set of tracks tracks a respective fish over the respective subset of time-ordered video frames.

13. The method of claim 1, wherein each track of the set of tracks tracks a set of key points in the head of a respective fish over the respective subset of time-ordered video frames, the set of key points comprising the tip of the upper lip of the head of the respective fish, the tip of the lower lip of the head of the respective fish, and the base of the jaw of the head of the respective fish.

14. The method of claim 1, further comprising: determining the average breathing rate based on a number of usable video frames across the set of tracks.

15. A system comprising: at least one processor; memory; and instructions stored in the memory to be executed by the at least one processor for performing a method as recited in any one of claims 1-14.

Images