A multi-modal fusion-based aquatic animal health early warning method

Abstract

The present application relates to the technical field of aquaculture, and more particularly to an aquaculture animal health early warning method based on multi-modal fusion. The sampling platform is raised at regular intervals to obtain sampling sample individuals, and after the image data of the samples are input into the recognition module, the recognition module will extract the motion trajectory and body surface features based on the input data, and then calculate the overall feeding condition, health condition, activity level and growth condition of the culture pond through the health early warning model. The structured state data of the previous sampling in the culture cycle is collected, and the environmental dissolved oxygen data obtained by the environmental perception part constitutes a time sequence as input, and the output is taken as the target, and the time sequence model is trained to learn the correlation function between the state evolution and the output. Overcome the problem of low accuracy of aquaculture animal health early warning in the process of monitoring aquatic products in the prior art due to the lack of correlation between the static characteristics, dynamic characteristics and dissolved oxygen content of aquatic products and the prediction of non-disease growth output of aquatic products.

Description

Technical Field

[0001] This invention relates to the field of aquaculture technology, and in particular to a method for early warning of aquatic animal health based on multimodal fusion. Background Technology

[0002] The rapid development of aquaculture has led to continuously increasing stocking densities and higher requirements for management precision. Timely and accurate monitoring of aquatic animal health is crucial for disease control, loss reduction, and ensuring output. However, aquatic animals live in water, making direct and continuous observation of their health difficult. Farm managers often only discover problems after obvious abnormalities appear, by which time irreversible economic losses have already occurred. More importantly, even if the aforementioned physical and behavioral characteristics can be obtained, existing monitoring paradigms remain at the level of isolated judgments of the current state, failing to establish a quantitative mapping relationship between the multi-dimensional characteristics continuously collected throughout the entire farming cycle and output indicators. Therefore, there is an urgent need in this field for a method that can predictively correlate the temporal evolution of individual static characteristics, dynamic characteristics, and dissolved oxygen levels with long-term output, providing highly accurate decision support for real-time monitoring of growth and refined aquaculture management.

[0003] Chinese patent publication number CN119534781A discloses a multimodal sensing technology method for monitoring aquaculture. It collects water body data such as water temperature and dissolved oxygen by integrating multiple sensors, and obtains a comprehensive water quality assessment result after data fusion and deep learning processing. However, its monitoring object is the aquatic environment rather than the aquatic products themselves, and it cannot capture the behavior of aquatic animals and early health signals on their body surface.

[0004] Chinese patent publication CN118672228A discloses a multi-parameter intelligent monitoring system and method for aquaculture based on the Internet of Things (IoT). This system analyzes multiple environmental parameters using a predictive model to classify the environmental state of the aquaculture area. Simultaneously, it acquires the behavioral states of aquatic animals through cameras and generates state scores. By combining environmental and behavioral states, it analyzes the overall state of the aquaculture area and generates control strategies to dynamically optimize monitoring intensity and management priorities. While this method introduces behavioral state scores, it only provides a general state assessment at the group level. It fails to quantify and extract multi-faceted characteristics of the aquatic animals themselves, and neither establishes a predictive correlation between these characteristics and output. Therefore, it is difficult to achieve accurate early warning and output prediction before diseases cause group abnormalities.

[0005] The existing technology still has the following problems: Monitoring remains primarily focused on the aquatic environment. Existing solutions either concentrate on multimodal fusion of water quality parameters such as water temperature and dissolved oxygen, or simply combine environmental and behavioral states. However, none of these solutions incorporate dissolved oxygen levels, individual body surface characteristics, and behavioral traits. Furthermore, they fail to establish quantitative predictive correlations between multidimensional health characteristics collected throughout the aquaculture process and output indicators, resulting in low accuracy in identifying non-pathological production anomalies. While management factors such as insufficient feed intake and nutritional imbalances may not cause obvious surface lesions, they can manifest through persistently low abdominal fullness and progressively decreasing activity levels. Existing solutions lack the ability to capture and quantify these latent signals, leading to low predictive accuracy for such problems. Summary of the Invention

[0006] To address this, the present invention provides a method for early warning of aquatic animal health based on multimodal fusion, which overcomes the problem in the existing technology of monitoring aquatic products in aquaculture where the lack of correlation between the static and dynamic characteristics of aquatic products, dissolved oxygen levels, and the prediction of non-pathological growth and output of aquatic products leads to variations in output prediction and results in low accuracy of early warning of aquatic animal health.

[0007] To achieve the above objectives, this invention provides a method for early warning of aquatic animal health based on multimodal fusion, comprising: Collect video data of live aquatic samples within a predetermined period in each aquaculture unit area; Based on the video data information combined with target detection and multi-target tracking algorithms, the motion trajectory of live aquatic samples is obtained; Based on the motion trajectory, dynamic behavioral characteristics of live aquatic samples are extracted; In response to the dynamic behavioral characteristics, high-resolution static image data of live aquatic samples are extracted; Analyze static body surface features based on the high-definition static image data; A target body health early warning model is constructed based on the static body surface features and the dynamic behavioral features of the corresponding live aquatic samples. The health level of the target body is determined based on the aforementioned health early warning model; Real-time detection of dissolved oxygen levels in the aquatic environment and recording of corresponding detection timestamps; Based on the aforementioned dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels, a training dataset is constructed by combining the output data information of the corresponding breeding units in each breeding cycle. Train a time series prediction model based on the aforementioned training dataset; Based on the time-series prediction model, the expected output fluctuations are predicted in real time by inputting the sampling status sequence of any aquaculture unit area to be predicted, and early warnings are issued based on the prediction results.

[0008] Furthermore, the video data information is a video stream of the activity of live aquatic samples collected from the water body by the sampling platform and captured by the image acquisition device over a period of time. The time period is the preset collection time length after the sampling platform extracts the live aquatic samples.

[0009] Furthermore, the process of obtaining the motion trajectory of live aquatic samples based on video data information combined with target detection and multi-target tracking algorithms includes: The video data information is input into the target detection model to identify the position of each object in each frame and output the bounding box; then the multi-object tracking algorithm assigns a unique identifier to each detected individual and associates the same identifier across consecutive frames to achieve continuous tracking across frames; From all tracked targets, the centroid pixel coordinates of samples are extracted frame by frame, converted into actual physical positions using pre-calibrated camera parameters, and a motion trajectory containing time series and spatial coordinates is formed.

[0010] Furthermore, the dynamic behavioral characteristics of the live aquatic samples include: jumping characteristics, tumbling characteristics, static state and collision situation, and the activity level of the sample group is calculated based on the above characteristics identification. Among them, the jumping feature recognition is obtained by analyzing the displacement change of each individual, the tumbling feature recognition is obtained by judging the coordinate difference of the same individual, the static state is obtained by analyzing the individual's continuous low speed and no autonomous movement within a time period, and the collision situation is eliminated by combining the movement of all individuals to exclude the misjudgment of inactive individuals due to collisions. The process of statistically analyzing the activity level of the sample group includes: counting the number of jumping individuals, the number of flipping individuals, and the number of stationary individuals, excluding accidental collisions, and calculating the proportion of each state to reflect the activity level of the sample group.

[0011] Furthermore, in response to dynamic behavioral characteristics, the process of extracting high-definition static image data information of live aquatic samples includes: identifying the static condition of each live aquatic sample from the acquired motion trajectory and activity characteristics; automatically selecting the optimal frame based on indicators such as image clarity, individual posture angle, and occlusion; cropping the individual region of the sample in the optimal frame using an instance segmentation mask; uploading it to the cloud; and generating static image data associated with the sample identifier after analysis by a visual language model, which serves as input for subsequent analysis.

[0012] Furthermore, the static body surface features include: the number of samples on the sampling platform, the individual length of each sample, the individual posture, the individual's feeding level, and the individual's body fullness.

[0013] Furthermore, the process of constructing a target body health early warning model based on the static body surface features and the corresponding dynamic behavioral features of live aquatic samples includes: for each live aquatic sample, aligning and splicing its static body surface features and dynamic behavioral features at the individual level to form a multidimensional health feature vector; summarizing the feature vectors of all samples within the same aquaculture unit and calculating the proportion of individuals with each health condition; organizing the statistical features of this sampling and the historical features of previous samplings of this unit according to time series and inputting them into the target body health early warning model constructed based on a temporal neural network; the model outputs the current health risk level, and when the prediction result triggers the preset early warning conditions, it generates and pushes early warning information.

[0014] Furthermore, the process of real-time detection of dissolved oxygen in the aquatic environment and recording the corresponding detection timestamp includes: deploying a fluorescence dissolved oxygen meter in the aquaculture water body, wherein the fluorescence dissolved oxygen meter monitors dissolved oxygen in real time for 24 hours, and records the timestamp of the current moment with each collection to form a corresponding time-series data stream.

[0015] Furthermore, the process of constructing a training dataset based on the dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels, in conjunction with the conditions of each breeding cycle, includes: summarizing the dynamic behavioral characteristics and static characteristics of all individuals in a single sampling of the same breeding unit into a group health snapshot, and aligning it with the mean and fluctuation index of dissolved oxygen levels during that period to generate a multimodal feature record for that sampling; organizing the records of each sampling in a complete breeding cycle into a feature sequence in chronological order, using the output index of a single breeding cycle as a label to form a training sample; and collecting samples from each breeding cycle and each breeding unit to form a training dataset for training the health early warning model.

[0016] Furthermore, based on the aforementioned time-series prediction model, the process of real-time prediction of expected output fluctuations from the sampled status sequence of any aquaculture unit area to be predicted, and issuing early warnings based on the prediction results, includes: for the current aquaculture unit, reading all structured data that have been sampled since the beginning of the current cycle from the database, constructing a multimodal feature sequence containing dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels in chronological order; inputting this sequence into the trained time-series prediction model, which infers and outputs predicted values; and simultaneously detecting whether there is a continuous deterioration trend in the future state. If any early warning condition is met, an early warning notification is generated and pushed to the aquaculture management terminal. The predicted values ​​include a future health status prediction sequence and output prediction values, and the early warning notification includes a risk level, estimated losses, and recommended measures.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for early warning of aquatic animal health based on multimodal fusion. By associating the static and dynamic characteristics of aquatic products with dissolved oxygen levels and the prediction of non-pathological growth and output of aquatic products, the health status of each sampling is time-seriested. The entire cycle of sampling sequences is linked with the final output indicators to train a large model, which directly predicts the final output. This directly opens up a quantitative channel between the dynamic state in the aquaculture process and the final economic output. It not only warns of problems that have already occurred, but also associates with the final output status, providing a more intuitive understanding of the aquaculture situation. This overcomes the problem in existing technologies where the lack of correlation between the static and dynamic characteristics of aquatic products and dissolved oxygen levels and the prediction of non-pathological growth and output of aquatic products leads to inconsistent output predictions and low accuracy of aquatic animal health early warning.

[0018] In particular, this invention enables the system to capture progressively deteriorating trends by serializing the entire cycle of sampling and linking it to output. When aquatic activity gradually declines, and feeding and growth deviate from predictions but no obvious organic lesions appear, the time-series prediction model can identify risks from the trends and trigger early warnings, rather than waiting for disease outbreaks or deaths to occur before issuing an alarm. This significantly advances the warning window, providing aquaculture managers with ample time to intervene and effectively solving the problem of delayed warnings caused by existing rule-based alarm systems that rely on already occurring anomalies.

[0019] In particular, this invention correlates the static and dynamic characteristics of aquatic products and dissolved oxygen levels obtained from various sampling inspections during the aquaculture process with output indicators of the aquaculture cycle. By training a time-series prediction model, a quantitative mapping channel is established from aquatic product growth data within the aquaculture cycle to the final actual output. Unlike existing technologies that can only determine whether the current state is normal, this invention can directly predict the survival rate and average weight at harvest for a single aquaculture cycle based on early and mid-term sampling sequences. This allows aquaculture managers to predict the harvest earlier and more accurately, improving the accuracy of aquatic animal health early warning. Attached Figure Description

[0020] Figure 1 This is a block diagram of the aquatic animal health early warning system based on multimodal fusion, as described in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the steps of the aquatic animal health early warning method based on multimodal fusion, as described in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0022] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] Please see Figure 1 This is a structural block diagram of a multimodal fusion aquatic animal health early warning system according to an embodiment of the present invention. The system described in the embodiment includes: The sampling module is used to automatically lift live aquatic samples from the aquaculture water at a preset time, triggering the top high-definition magnifying glass-level camera to record a short video, and to obtain standardized video data information in a low-stress manner. The video information is captured by a high-definition camera installed on the sampling platform; The environmental sensing module uses a fluorescent dissolved oxygen meter deployed in the aquaculture water to collect dissolved oxygen concentrations at a preset frequency around the clock and record timestamps simultaneously. It extracts data that overlaps with the video recording period and calculates the average and fluctuation range of dissolved oxygen during that period. The target detection module, which is connected to the sampling module, is used to input video data information frame by frame into the target detection model to identify the bounding box of each sample individual. A multi-target tracking algorithm is used to assign a unique identifier to each individual and maintain cross-frame correlation. After camera calibration conversion, the continuous spatiotemporal motion trajectory of each sample is generated. The dynamic behavior feature extraction module, which is connected to the target detection module, is used to extract dynamic behavior features based on motion trajectory. It identifies jumping events by calculating the centroid displacement between adjacent frames, detects tumbling behavior by analyzing the rate of change of posture, determines the stationary state by using a low speed threshold, and detects collisions by combining individual spacing. The static body surface feature extraction module, which is connected to the target detection module, is used to extract high-definition static images of the optimal posture from the video based on the motion trajectory and extract static body surface features, including individual length, posture classification, abdominal fullness and feeding degree. The health level determination module is connected to the dynamic behavior feature extraction module and the static body surface feature extraction module respectively. It is used to align and concatenate the dynamic behavior features and static body surface features of the sample at the individual level to form a multi-dimensional feature vector. Through joint decision-making by the rule engine and the classification model, the health level of each individual is output. The four indicators of dynamic behavior features are weighted and fused to determine the activity index of the group. The prediction and early warning module is connected to the health level determination module and the environmental sensing module respectively. It is used to receive the sampling status sequence of any breeding unit to be predicted, call the trained time series prediction model to output the future status prediction and output prediction values ​​in real time, trigger graded early warning through threshold comparison and deterioration trend dual detection, and push early warning notification containing risk level and suggested measures to the management terminal. The time-series prediction model is trained using acquired dynamic behavioral features, static body surface features, and dissolved oxygen levels, and is put into use after training is completed.

[0024] Please see Figure 2 This is a flowchart illustrating the steps of a multimodal fusion-based aquatic animal health early warning method according to an embodiment of the present invention. The method described in the embodiment includes: Step S1: Collect video data of live aquatic samples within a predetermined period in each aquaculture unit area; based on the video data and combined with target detection and multi-target tracking algorithms, obtain the movement trajectory of the live aquatic samples; Step S2: Based on the motion trajectory, extract the dynamic behavioral features of the live aquatic samples; in response to the dynamic behavioral features, extract high-resolution static image data information of the live aquatic samples. Step S3: Analyze the static body surface features based on the high-definition static image data; construct a target body health early warning model based on the static body surface features and the dynamic behavioral features of the corresponding live aquatic samples; determine the health level of the target body based on the health early warning model; Step S4: Real-time detection of dissolved oxygen in the aquatic environment and recording of the corresponding detection timestamp; Step S5: Based on the dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels, construct a training dataset in conjunction with the conditions of each breeding cycle; train a time-series prediction model based on the dataset. Step S6: Based on the time-series prediction model, input the sampling status sequence of any aquaculture unit area to be predicted to predict the expected output fluctuation in real time, and issue an early warning based on the prediction results.

[0025] Specifically, the division of the aquaculture cycle in this embodiment is based on the stocking date and the harvest date. The embodiment includes: a complete aquaculture cycle begins from the date the same batch of aquatic products is stocked into the aquaculture unit and ends on the date that batch of aquatic products is harvested. Taking shrimp pond aquaculture as an example, the cycle officially begins when shrimp larvae are stocked into the aquaculture unit and continues until the shrimp reach marketable size, at which point they can be harvested, and the aquaculture cycle ends.

[0026] Specifically, the division of aquaculture units in the embodiment is based on independent aquaculture ponds or net cages. The embodiment includes ponds or net cages in which the same batch of seedlings are separately stocked and fed, monitored and harvested separately throughout the entire process, which constitute an independent aquaculture unit with independent feeding management, water quality control and image acquisition devices. Specifically, the video data information refers to the live aquatic samples extracted from the water body by the sampling platform, and the video stream of the sample activity over a period of time captured by the image acquisition device. The time period is the preset collection time length after the live aquatic sample is extracted by the sampling platform.

[0027] In this embodiment, the sampling platform retrieves a predetermined number of live aquatic samples from the aquaculture water at a preset sampling time. The sampling platform is an automatic lifting bait net. After the live aquatic samples are retrieved, an image acquisition device, including a high-definition magnifying glass, a camera, and a supplementary light, is fixedly installed above the observation container. After the live aquatic samples are lifted, the system automatically starts the image acquisition device to begin recording. The recording time is a preset acquisition time, such as 3 to 5 seconds. During the recording, the live aquatic samples move freely in the container, and the image acquisition device continuously captures their movements. After the recording is completed, the resulting video stream is the video data information. The use of an automatic lifting bait net can reduce the stress of retrieval, making the captured images closer to the actual healthy state. The high-definition magnifying glass-level camera can clearly capture microscopic details on the body surface, and standardized shooting conditions ensure the consistency of data from different batches, providing a high-quality image foundation for subsequent feature extraction and health assessment.

[0028] In this embodiment, based on the actual decision-making needs of aquaculture, an intelligent feeding net is used to automatically complete sample lifting and imaging at the time of feeding. The early warning notification directly outputs the risk level, estimated loss and suggested measures, allowing managers to make quick decisions, which greatly improves the efficiency of aquaculture management, reduces labor consumption and shortens the decision response time. At the same time, the behavioral performance, body surface condition and environmental data of each sampling are treated as a whole for multi-dimensional cross-validation. The system can autonomously identify early risk signals from the evolution of the situation, automatically filter out occasional fluctuations and environmental interference, and predict the degree of impact on survival rate and average weight at harvest before health risks show obvious symptoms, thus improving the accuracy of early warning.

[0029] Specifically, the process of obtaining the motion trajectory of live aquatic samples based on video data information combined with target detection and multi-target tracking algorithms includes: The video data is input into the target detection model to identify the position of each object in each frame and output bounding boxes; then the multi-object tracking algorithm assigns a unique identifier to each detected individual and associates the same identifier across consecutive frames to achieve continuous tracking across frames; From all tracked targets, the centroid pixel coordinates of samples are extracted frame by frame, converted into actual physical positions using pre-calibrated camera parameters, and a motion trajectory containing time series and spatial coordinates is formed.

[0030] In this embodiment, the video data is input frame by frame into a trained target detection model. The target detection model uses a YOLOv8 network architecture to locate individual targets in each frame, outputting the bounding box coordinates and detection confidence scores for each target. Then, the DeepSORT multi-target tracking algorithm is used to assign a unique identifier (ID) to each newly detected target. By fusing the target's appearance and motion features, a stable association of the same ID is maintained between consecutive frames, achieving continuous tracking across frames. When a target is lost due to brief occlusion, its location is predicted and completed using Kalman filtering.

[0031] From all successfully tracked targets, extract the centroid pixel coordinates of each sample frame by frame. The centroid is obtained by calculating the geometric center from the bounding box vertex coordinates. Arrange the centroid coordinates of the same ID in each frame in chronological order, and convert the pixel coordinates into actual physical coordinates within the observation container using pre-calibrated camera intrinsic and extrinsic parameter matrices, forming a motion trajectory containing a timestamp sequence and spatial coordinates.

[0032] In this embodiment, individualized and precise motion capture is achieved for each sample. In a live sampling scenario, individuals move freely, frequently intersecting and briefly occluding each other, with random movement directions. A unique identifier is assigned to and maintained for each sample. Even with brief occlusion, trajectory completion can be achieved through motion prediction, ensuring that each sample's motion trajectory is independent and complete, overcoming the problem of individual data confusion in multi-target scenarios. Simultaneously, after frame-by-frame centroid coordinates are converted to actual physical coordinates through camera calibration, the resulting spatiotemporal trajectory sequence has a realistic scale, and sampling data from different batches and time points are highly comparable. This design provides a standardized, high-resolution individual-level motion data foundation for subsequent dynamic behavioral feature extraction, significantly improving the accuracy and reliability of motion data extraction.

[0033] Specifically, the dynamic behavioral characteristics of the live aquatic samples include: jumping characteristics, tumbling characteristics, static state and collision situation, and the activity level of the sample group is calculated based on the above characteristics. Among them, the jumping feature recognition is obtained by analyzing the displacement change of each individual, the tumbling feature recognition is obtained by judging the coordinate difference of the same individual, the static state is obtained by analyzing the individual's continuous low speed and no autonomous movement within a time period, and the collision situation is eliminated by combining the movement of all individuals to exclude the misjudgment of inactive individuals due to collisions. The process of statistically analyzing the activity level of the sample group includes: counting the number of jumping individuals, the number of flipping individuals, and the number of stationary individuals, excluding accidental collisions, and calculating the proportion of each state to reflect the activity level of the sample group.

[0034] Specifically, in response to dynamic behavioral characteristics, the process of extracting high-definition static image data of live aquatic samples includes: identifying the static condition of each live aquatic sample from the acquired motion trajectory and activity characteristics; automatically selecting the optimal frame based on indicators such as image clarity, individual posture angle, and occlusion; cropping the individual region of the sample in the optimal frame using an instance segmentation mask; uploading it to the cloud; and generating static image data associated with the sample identifier after analysis by a visual language model, which serves as input for subsequent analysis.

[0035] In this embodiment, the jumping feature is obtained by calculating the vertical displacement of the centroid between adjacent frames. When the displacement exceeds a preset threshold within 0.1 seconds, it is determined as a jumping event, and the jumping frequency per unit time is counted. The tumbling feature is identified by analyzing the rate of change of the angle between the individual's major axis and the horizontal plane. When the angle suddenly exceeds the threshold and a belly-up posture occurs, the tumbling event and its duration are recorded. The stationary state is determined by the centroid displacement rate being consistently below 0.5 cm / s and the absence of voluntary movement. The proportion of the stationary period to the total duration is counted. The collision situation is based on the results of multi-target tracking, calculating the Euclidean distance between the centroids of individuals in real time. When the distance is less than the threshold and persists for a certain number of frames, the collision event and its frequency are recorded. After normalizing the above four indicators, a weighted fusion is performed, and the mean of the scores of all samples is taken and corrected by the proportion of abnormal individuals. The output is a group activity index from 0% to 100%. The lower the score, the worse the overall activity, indicating potential health risks.

[0036] In this embodiment, the activity level that traditionally relies on subjective human judgment is transformed into four quantifiable behavioral indicators. By using multi-dimensional weighted fusion, misjudgments caused by fluctuations in a single indicator are eliminated. Furthermore, the correction factor ensures that the group activity index can reflect the overall average level while remaining sensitive to a few abnormal individuals. Early warning signals can be captured through behavioral abnormalities in the early stages of disease or stress, thus improving the timeliness and efficiency of early detection of health risks.

[0037] Specifically, the static body surface features include: the number of samples on the sampling platform, the individual length of each sample, the individual posture, the individual's feeding level, and the individual's body fullness.

[0038] In this embodiment, shrimp samples were used for detection. Static surface features of the samples were extracted using a VLM inference model. These static surface features included the number of samples on the sampling platform, the individual length of each sample, individual posture, individual feeding level, and individual body fullness. The extraction process was as follows: The number of samples was directly obtained by counting the number of valid detection boxes output by the target detection model. Individual length was calculated by converting pixel length to actual physical length in millimeters based on the bounding box dimensions of each sample and camera calibration parameters. Individual posture was determined by locating the head, tail, and dorsal base points using a keypoint detection model, calculating the angle between the body's long axis and the horizontal plane, and classifying postures such as lateral and supine. Individual feeding level was assessed by analyzing the clarity of the shrimp vein; a clear shrimp vein indicated normal feeding, an unclear shrimp vein indicated insufficient feeding, and an indistinct shrimp vein indicated no feeding. The fullness of an individual's body is assessed by determining whether the shrimp's body is full and whether there is a cavity in its tail. The results of the fullness of the individual's body and the characteristics of the empty tail are compared with the corresponding preset standards for fullness of the individual's body and characteristics of the empty tail, respectively, to determine whether the feeding situation is normal.

[0039] In this embodiment, traditional experience is transformed into quantifiable and recordable standardized indicators, eliminating the subjective differences of manual visual inspection. Individual feeding levels are determined through dynamic comparison, objectively reflecting feeding status without relying on feeding records. The five features comprehensively reflect the static body surface state of each sample from five dimensions: quantity, size, body shape, feeding, and appearance. This provides multi-dimensional and highly consistent reliable data input for subsequent health level determination and time series model training, improving the objectivity and accuracy of health assessment.

[0040] Specifically, the process of constructing a target body health early warning model based on the static body surface features and the corresponding dynamic behavioral features of live aquatic samples includes: for each live aquatic sample, aligning and splicing its static body surface features and dynamic behavioral features at the individual level to form a multidimensional health feature vector; summarizing the feature vectors of all samples within the same aquaculture unit and calculating the proportion of individuals with each health condition; organizing the statistical features of this sampling and the historical features of previous samplings of this unit according to time series and inputting them into the target body health early warning model constructed based on a temporal neural network; the health early warning model outputs the current health risk level, and when the prediction result triggers the preset early warning conditions, it generates and pushes early warning information.

[0041] In this embodiment, for each live aquatic sample, its static body surface features and dynamic behavioral features are aligned and spliced ​​at the individual level according to the sample ID, forming a multi-dimensional health feature vector containing sample quantity, individual length, individual posture, feeding level, body fullness, jumping frequency, turning frequency, static duration percentage, and collision frequency. The feature vectors of all samples within the same aquaculture unit are summarized, and the number and percentage of individuals in each health condition are statistically analyzed according to preset health judgment rules. Health conditions are divided into four levels: healthy, sub-healthy, obviously abnormal, and severely near death. The statistical features of the sampled population and the historical statistical features of previous samplings in the same unit are organized into a time series to form a sampling status sequence. This sequence is input into a temporal neural network model based on the Transformer architecture, and a health early warning model capable of application is trained based on this model. The health early warning model outputs the current health risk level and future trend prediction. When the prediction results show an increase in the health risk level or a continuous decrease in the population activity index and a continuous increase in the percentage of abnormal individuals, preset early warning conditions are triggered, generating early warning information including risk level, abnormal attribution, and suggested measures, and pushing it to the management terminal.

[0042] In this embodiment, the isolated assessment of a single sampling inspection is upgraded to a multi-dimensional dynamic early warning based on time-series evolution. The health early warning model not only focuses on the current state but also captures the continuous changing trend of the health status, providing early warning before the anomaly develops into a large-scale outbreak. At the same time, individual-level feature alignment ensures that the sample data can locate the performance of specific abnormal individuals. The combination of group statistics and individual characteristics enables the early warning to reflect the overall risk, facilitate accurate decision-making by managers, effectively improve the timeliness of early warning, and save management costs.

[0043] Specifically, the process of constructing a training dataset based on the dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels, in conjunction with the situation of each breeding cycle, includes: summarizing the dynamic behavioral characteristics and static characteristics of all individuals in a single sampling of the same breeding unit into a group health snapshot, and aligning it with the mean and fluctuation index of dissolved oxygen levels during that period to generate a multimodal feature record for that sampling; organizing the records of each sampling in a complete breeding cycle into a feature sequence according to time sequence, and using the output index of a single breeding cycle as a label to form a training sample.

[0044] The process of training the time series prediction model based on the training dataset in this embodiment is as follows: Each multi-dimensional feature sequence in the constructed training dataset is used as input, and the corresponding output index is used as the supervision label to perform end-to-end supervised training on the time series prediction model. The time series prediction model adopts a Transformer-based encoder architecture, which includes a multi-head self-attention layer and a feedforward fully connected layer. The self-attention mechanism captures long-distance dependencies between different sampling time steps. The training configuration is as follows: mean squared error is used as the loss function, the Adam optimizer is used, the initial learning rate is set to 0.001, the batch size is 32, the training epochs are no less than 100, and an early stopping mechanism is set to terminate training when the validation set loss does not decrease for 10 consecutive epochs to prevent overfitting. After training, the time series prediction model parameters are fixed and saved as the model file, and deployed to an online inference environment. The training dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio. The training set is used for updating the time series prediction model parameters, the validation set is used for early stopping judgment and hyperparameter selection, and the test set is used to evaluate the generalization performance of the time series prediction model.

[0045] In this embodiment, the Transformer's self-attention mechanism can automatically learn the differentiated impact weights of different sampling time steps on the output. Compared with traditional recurrent neural networks, it is better at capturing long-distance dependencies, enabling the model to make reliable predictions of mid-to-late-stage output using early-stage aquaculture data. The end-to-end training method eliminates the need for manually defining intermediate rules, and the time-series prediction model directly learns the mapping relationship between aquatic growth data and output data within the aquaculture cycle from the data. The early cessation mechanism and dataset partitioning strategy effectively suppress overfitting, enabling the model to maintain stable prediction accuracy on unseen aquaculture cycles.

[0046] Specifically, the process of real-time detection of dissolved oxygen in the aquatic environment and recording the corresponding detection timestamp includes: deploying a fluorescence dissolved oxygen meter in the aquaculture water body, wherein the fluorescence dissolved oxygen meter monitors dissolved oxygen in real time for 24 hours, and records the timestamp of the current moment with each collection to form a corresponding time-series data stream; In this embodiment, a fluorescence-based dissolved oxygen meter is deployed in the middle layer of the aquaculture pond water, 10 to 15 centimeters below the water surface. This location is the main water layer where the feed net is set, and can represent the actual dissolved oxygen environment of the shrimp population. The fluorescence-based dissolved oxygen meter is based on the principle of fluorescence quenching. It calculates the dissolved oxygen concentration by measuring the fluorescence lifetime of the fluorescent material on the sensitive membrane surface after excitation. Compared with the traditional electrochemical method, it has the advantages of fast response, no need for stirring, and low maintenance. The dissolved oxygen meter continuously collects the dissolved oxygen concentration value of the water 24 hours a day at a frequency of once every 5 minutes, with a measurement range of 0 to 20 mg / L and an accuracy of ±0.1 mg / L, expressed in milligrams per liter, accurate to one decimal place. Each collection simultaneously records the current time timestamp, accurate to the second. The collected dissolved oxygen levels and timestamps are uploaded to the server database in real time via an IoT gateway, forming a continuous time-series data stream of dissolved oxygen. When the sampling inspection is completed, the system automatically extracts the dissolved oxygen records that overlap with the video recording time from the data stream, calculates the average dissolved oxygen level and fluctuation range of that time period, and stores them as environmental characteristic data, behavioral characteristics and body surface characteristics of the current sampling inspection in a time-aligned and correlated manner.

[0047] In this embodiment, the fluorescence dissolved oxygen meter does not require oxygen consumption for measurement, making it particularly suitable for low-flow-rate or stagnant water environments. 24-hour high-frequency continuous acquisition can completely capture instantaneous fluctuations in dissolved oxygen during key periods such as before and after feeding, and day-night transitions, avoiding the omission of short-term hypoxia events due to excessively large sampling intervals. Aligning dissolved oxygen levels with sampling data time enables the system to distinguish between abnormal behavior caused by water hypoxia and abnormal behavior caused by disease, effectively reducing the false alarm rate caused by environmental factors.

[0048] Specifically, the accumulated sampling status sequence of any ongoing aquaculture unit since its inception is input into a pre-trained time-series prediction model to predict subsequent status and expected output in real time, and to issue early warnings based on the prediction results. For the current aquaculture unit, all structured data that has been sampled since the beginning of the current cycle is read from the database, and a multimodal feature sequence containing dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels is constructed in chronological order. This sequence is input into the pre-trained time-series prediction model, which infers and outputs predicted values. At the same time, it detects whether there is a continuous deterioration trend in the future status. If any early warning condition is met, an early warning notification is generated and pushed to the aquaculture management terminal.

[0049] In this embodiment, the process of inputting the accumulated sampling status sequence of any ongoing aquaculture unit since its inception into a pre-trained time-series prediction model to predict subsequent status and expected output in real time, and issuing early warnings based on the prediction results, is as follows: For the current aquaculture unit, all structured data of completed sampling since the start of the current aquaculture cycle is read from the server database, and a multimodal feature sequence containing dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen is constructed in chronological order, with each time step corresponding to a complete feature vector of one sampling. This sequence is input into the pre-trained time-series prediction model, which outputs two sets of predicted values ​​through forward propagation inference: one set is a health status prediction sequence for several future samplings, including the trend of the population activity index and the changing trend of the proportion of each health level; the other set is the output prediction value, including the survival rate prediction value and the average weight at harvest. The early warning judgment adopts a dual mechanism: the output prediction value is compared with a preset grading threshold, and at the same time, it is detected whether the health status prediction sequence shows a trend of two consecutive declines in the population activity index and an increase in the proportion of abnormal individuals. If the conditions are met, a trend deterioration warning is triggered. When any of the early warning conditions are met, the system automatically generates an early warning notification. The notification includes the risk level, the estimated loss range, and suggested intervention measures, and is pushed to the management terminal through the aquaculture management APP or platform message interface.

[0050] In this embodiment, the early warning triggering mechanism is upgraded from a single threshold static judgment to a dual prediction of output analysis and deterioration trend. This not only prevents the macro risks of substandard output, but also captures the micro signals of continuous deterioration of the process state, filling the gap in the difficulty of predicting gradual risks. At the same time, the time series prediction model re-infers and refreshes the prediction results after receiving new sampling data each time, which improves the prediction accuracy.

[0051] Specifically, the predicted values ​​include a future health status prediction sequence and output prediction values, and the early warning notification includes a risk level, estimated losses, and recommended measures.

[0052] In this embodiment, the predicted values ​​are displayed to aquaculture managers via a real-time overview interface on the user terminal. Different interfaces display the latest core indicators of each aquaculture unit, including: a real-time display of the group activity index with values ​​from 0% to 100% and corresponding color progress bars (green for low activity, red for high activity); the movement states of different individuals, such as jumping, turning over, and remaining still, marked with different colors; and the real-time display of the number of state triggers. Simultaneously, it identifies and displays independent data for each live aquatic sample by sampling batch, including: individual number, high-resolution static image, body length, posture classification, feeding level, and shrimp vein clarity. This data is compared with historical sampling data to show the current aquaculture stage and changes in growth. A list of triggered warning notifications is displayed in reverse chronological order. Each notification includes: warning level, trigger time, predicted survival rate and average weight at harvest, estimated yield loss and economic loss, main attribution characteristics, and suggested measures, providing farmers with intuitive data support.

[0053] In this embodiment, the predicted values ​​output by the time-series prediction model are presented in a visual manner through the user terminal, allowing aquaculture managers to intuitively grasp the health status and future risk trends of the fish population without having to view the raw data. The real-time overview interface displays the population activity index using numerical values ​​combined with color progress bars, transforming complex behavioral data into easily understandable visual information; the historical trend interface facilitates managers to compare the current status with the expected deviation; the early warning notification directly outputs the risk level, predicted survival rate and average weight at harvest, estimated losses, and suggested measures, enabling rapid decision-making without relying on secondary analysis based on experience; the intuitive design significantly lowers the technical barrier to entry, significantly shortens the time from data acquisition to decision response, and achieves simplification of aquaculture management operations and a substantial improvement in decision-making efficiency.

[0054] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for monitoring and early warning of aquatic health based on multimodal fusion, characterized in that, include: Collect video data of live aquatic samples within a predetermined period in each aquaculture unit area; Based on the video data information combined with target detection and multi-target tracking algorithms, the motion trajectory of live aquatic samples is obtained; Based on the motion trajectory, dynamic behavioral features of live aquatic samples are extracted, including jumping features, tumbling features, static state, and collision situation. Among them, the jumping feature is obtained by calculating the vertical displacement of the centroid between adjacent frames, the flipping feature is identified by analyzing the rate of change of the angle between the individual's major axis and the horizontal plane, the stationary state is determined by the centroid displacement rate being consistently lower than the preset value and the absence of autonomous movement, and the collision situation is detected by comparing the Euclidean distance between the centroids of individuals with the preset threshold. In response to the dynamic behavioral characteristics, high-resolution static image data of live aquatic samples are extracted; Analyze static body surface features based on the high-definition static image data; From the acquired motion trajectories and activity characteristics, the static conditions of each live aquatic sample are identified, and the optimal frame is automatically selected; After cropping the individual region of the sample in the best frame, it is uploaded to the cloud and analyzed by a visual language model to generate static image data associated with the sample identifier, which serves as input for subsequent analysis of static body surface features. The static body surface features include the number of samples, the individual length of each sample, the individual posture, the individual's feeding level, and the individual's body fullness. Among them, the individual length is obtained by combining the bounding box size with the camera calibration parameters; the individual pose is determined by locating the head and tail endpoints through the key point detection model and calculating the angle between the individual's long axis and the horizontal plane; the individual's feeding degree is evaluated by comparing the morphology of the shrimp vein area of ​​the sample during the video period; and the individual's body fullness is graded by calculating the ratio and comparing it with the preset standard curve. A target body health early warning model is constructed based on the static body surface features and the dynamic behavioral features of the corresponding live aquatic samples. For each live aquatic sample, its static body surface features and dynamic behavioral features are aligned and spliced ​​at the individual level to form a multidimensional health feature vector. Summarize the feature vectors of all samples within the same breeding unit and count the proportion of individuals at each health level. The statistical characteristics of the sampling and the historical characteristics of previous samplings in this unit are organized in time series and input into the target body health early warning model based on the time series neural network, and the current health risk level is output. The health level of the target body is determined based on the aforementioned health early warning model; Real-time detection of dissolved oxygen levels in the aquatic environment and recording of corresponding detection timestamps; Based on the aforementioned dynamic behavioral characteristics, static body surface characteristics, and dissolved oxygen levels, a training dataset is constructed by combining the output data information of the corresponding breeding units in each breeding cycle. The dynamic behavioral characteristics and static body surface characteristics of all individuals in a single sampling of the same breeding unit are summarized into a group health snapshot, and aligned with the mean and fluctuation index of dissolved oxygen during that period to generate a multimodal characteristic record of the sampling. The records of each sampling inspection within a complete breeding cycle are organized into a feature sequence according to time sequence, and the output indicators of a single breeding cycle are used as supervision labels to form training samples. The training dataset consists of samples from a complete breeding cycle and several breeding units. Train a time series prediction model based on the aforementioned training dataset; Based on the time-series prediction model, the expected output fluctuations are predicted in real time by inputting the sampling status sequence of any aquaculture unit area to be predicted, and early warnings are issued based on the prediction results.

2. The aquatic health monitoring and early warning method based on multimodal fusion according to claim 1, characterized in that, The process of obtaining the movement trajectory of live aquatic samples includes: Obtain video data of live aquatic samples within a designated aquaculture unit area within a predetermined time period; The video data is input frame by frame into the target detection model, and the bounding boxes of the sample targets in each frame are output. A multi-target tracking algorithm is used to assign a unique identifier to each detected individual, and the same identifier is associated between consecutive frames to achieve continuous tracking across frames; The centroid pixel coordinates of samples are extracted frame by frame from all tracked targets, and converted into actual physical coordinates using pre-calibrated camera parameters to form a motion trajectory that includes time series and spatial coordinates.

3. The aquatic health monitoring and early warning method based on multimodal fusion according to claim 2, characterized in that, The process of real-time detection of dissolved oxygen levels in aquatic environments and recording corresponding timestamps includes: A fluorescence dissolved oxygen meter is deployed in the aquaculture water body to continuously collect dissolved oxygen concentration values ​​in the water body for 24 hours at a preset frequency. The timestamp of the current moment is recorded at the same time for each collection, forming a continuous time-series data stream of dissolved oxygen. When the sampling inspection is completed, the dissolved oxygen content records that overlap with the video recording time period are extracted from the data stream, and the average dissolved oxygen content and fluctuation range are calculated as environmental characteristic data for that sampling inspection.

4. The aquatic health monitoring and early warning method based on multimodal fusion according to claim 3, characterized in that, The process of training a time series prediction model based on the aforementioned training dataset includes: The data produced from the live aquatic samples include survival rate, average weight at harvest, and feed conversion ratio. The time-series prediction model is a time-series neural network that learns the weights of the differentiated impact of different sampling time steps on the output through a self-attention mechanism.

5. The aquatic health monitoring and early warning method based on multimodal fusion according to claim 4, characterized in that, The process of issuing early warnings based on prediction results includes: The accumulated sampling status sequence is input into the trained time series prediction model, which outputs the future health status prediction sequence and output prediction value. The predicted output value is compared with the preset grading threshold, and the system detects whether there is a continuous deterioration trend in the future health status prediction sequence. When any warning condition is met, a warning notification is generated and pushed to the management terminal.

6. The aquatic health monitoring and early warning method based on multimodal fusion according to claim 5, characterized in that, The future health status prediction sequence and output prediction value include the future health status prediction sequence of the breeding unit and the final output prediction value of the target breeding unit.

Images