A multi-modal based fish disease grading evaluation and triage method

Abstract

The application belongs to the technical field of machine learning, and discloses a fish disease grading evaluation and triage method based on multi-modalities. A multi-source data acquisition terminal array is constructed and uniformly deployed according to the breeding density. The terminal integrates an underwater high-definition binocular camera, a visible and near-infrared hyperspectral imaging device, a non-contact non-invasive physiological sensing unit and a multi-parameter water quality sensing unit. Three types of core modal data, i.e. visual, physiological and environmental data, are synchronously collected. Millisecond-level space-time alignment and metadata label annotation are completed. Modal adaptation preprocessing is completed through meta-learning. Then, a standardized and scene-adaptive multi-modality data set is constructed. Isolated forest algorithm optimized by meta-learning is used to complete abnormal data elimination, feature enhancement, data repair and standardization processing. Improved ResNet50, 3D-CNN and bidirectional LSTM networks are used to extract modal features. Core disease features are screened through correlation analysis, redundant feature elimination and dynamic weight distribution.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically a multimodal method for grading, assessing, and triaging fish lesions. Background Technology

[0002] Current fish disease assessment and triage technologies are mainly divided into three categories: traditional manual discrimination, single-modality intelligent detection, and laboratory biochemical diagnosis. Existing intelligent solutions have the following technical problems in the application of machine learning technology, and cannot meet the precise prevention and control needs of large-scale aquaculture. Existing solutions mostly use a single visual modality or water quality sensor data to build machine learning models, which belong to a single feature and single task learning architecture. They can only identify visible lesions on the fish body surface or abnormal water parameters, but cannot capture the fluctuations in physiological indicators caused by internal lesions, nor can they explore the correlation patterns between environmental, physiological, and surface behaviors. For early-stage fish lesions with only subtle abnormalities in physiological indicators and no visible surface features, existing models have low recognition accuracy and are prone to missing the best prevention and control window.

[0003] Current lesion grading technologies rely on fixed rules established through human experience, failing to develop a standardized grading system based on large-scale labeled data and optimized through machine learning. Inconsistent grading standards across different regions and aquaculture entities lead to inconsistent quality of labeled data used for model training, resulting in poor consistency and repeatability of grading results. Furthermore, existing solutions often employ single-classification learning models, which are inadequate for handling complex scenarios involving multiple pathogen infections and concurrent lesions in fish, resulting in low accuracy in matching grading to lesion types.

[0004] Existing triage and treatment technologies only match fixed treatment plans based on the graded results, without using machine learning methods such as reinforcement learning to build a dynamic decision-making mechanism. They cannot combine the real-time environment of the aquaculture scenario, the risk of disease transmission, the growth characteristics of fish species, and the feedback of treatment effects to optimize the triage strategy, which can easily lead to problems such as excessive drug residues due to overtreatment and disease spread due to undertreatment. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal method for the grading, assessment, and triage of fish lesions, in order to solve one or more of the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal method for grading and triaging fish lesions, comprising the following specific steps: Furthermore, in the data acquisition synchronization phase, a multi-source data acquisition terminal array is constructed and uniformly deployed in the aquaculture water according to the stocking density. The terminal integrates an underwater high-definition binocular camera, visible and near-infrared hyperspectral imaging equipment, a non-contact non-invasive physiological sensing unit, and a multi-parameter water quality sensing unit to achieve synchronous acquisition of three core modal data: visual, physiological, and environmental. The visual modal data includes fish surface lesion characteristics and swimming behavior temporal characteristics. The physiological modal data is collected through non-contact photoplethysmography visual sensing and non-contact surface impedance non-invasive detection technology to collect fish respiratory rate, heart rate, body surface temperature, and other physiological indicators. The environmental modal data collects environmental parameters such as dissolved oxygen, pH value, and ammonia nitrogen in the water. Using the unified clock of the acquisition terminal as a reference, the three types of modal data in the same time window and the same spatial area are spatiotemporally aligned at the millisecond level. Metadata tags are added to each set of aligned data. The tag content includes the farmed fish species, growth cycle, farming mode, farming density, and geographical area. Meta-learning-based modal adaptation preprocessing enables the acquisition terminal to adaptively match the physiological characteristics differences of different fish species and the fluctuations of the breeding environment through a small number of pre-trained samples, thereby filtering out invalid data acquisition scenarios in advance and constructing a standardized, scenario-adaptive multimodal dataset.

[0007] Furthermore, the data preprocessing stage is based on the standardized, scene-adaptive multimodal dataset constructed in the data acquisition and synchronization stage. For the three types of modal data, namely visual, physiological and environmental, the isolated forest algorithm based on meta-learning optimization is adopted. Through transfer learning, it adapts to the data fluctuation characteristics of different fish species and different aquaculture scenarios, and identifies and removes outliers and outliers caused by sensor interference, water body obstruction and poor equipment contact. For visual modal data, an image denoising algorithm based on Vision Transformer is used to remove water interference noise. An adaptive enhancement algorithm is used to highlight the features of lesion areas, and contrastive learning is used to optimize the feature enhancement effect, thereby strengthening the identification of weak features of early lesions. For physiological modal time-series data, an LSTM smoothing filtering algorithm is used to preserve the abnormal fluctuation trend of physiological indicators, and an attention mechanism is used to weight and highlight subtle fluctuation features related to lesions. For environmental modal time-series data, a Kalman filter algorithm is used to correct sensor drift errors and complete data benchmark calibration. A time-series prediction model is used to predict sensor drift trends in advance. To address short-term data gaps during data acquisition, a generative adversarial network is used to construct a time-series data repair model. Based on associated modal data within the same time period, the missing single-modal data is intelligently repaired. The min-max normalization algorithm and Z-score normalization algorithm are used to unify all data from the three modalities into the same dimensional range.

[0008] Furthermore, the feature extraction and screening stage is based on the standardized data output from the data preprocessing stage. For visual modality image and video data, a pre-trained improved ResNet50 convolutional neural network is used to extract spatial features, and a 3D-CNN is used to extract behavioral temporal action features. These are fused to generate a visual modality feature matrix. The network parameters are fine-tuned through meta-learning, enabling the feature extraction model to quickly adapt to the differences in body surface features of newly added fish species. For physiological modality and environmental modality temporal data, a bidirectional LSTM network is used to extract dynamic features such as fluctuation trends and mutation amplitudes of the temporal data, generating physiological modality and environmental modality feature matrices respectively. The Transformer attention mechanism is combined to enhance the extraction effect of key features. Based on the mutual information algorithm and Pearson correlation coefficient, the correlation between each extracted feature and the occurrence and development of fish lesions is calculated, and the contribution of each feature to lesion identification and classification is quantified. A correlation threshold is set, and redundant and noisy features with correlation below the threshold are removed. For the retained high-contribution features, the initial weights are dynamically assigned through reinforcement learning. The weights are adaptively adjusted according to the feature importance of different fish species and different lesion types to generate a filtered feature set of three modalities.

[0009] Furthermore, the cross-modal fusion stage, based on the three types of modal filtered feature sets generated in the feature extraction and filtering stage, adopts a meta-learning optimized cross-attention hierarchical fusion model. The fusion model is divided into three fusion levels, which sequentially complete intra-modal feature fusion, cross-modal pairwise feature fusion, and global cross-modal deep fusion. For the filtered feature sets of the visual, physiological, and environmental modalities, a self-attention mechanism is adopted to complete the fusion of different features within the same modality, generating single-modal fusion feature vectors for the three modalities respectively. A bidirectional cross-attention mechanism is adopted to complete the pairwise cross-modal feature fusion of vision and physiology, physiology and environment, and vision and environment. Reinforcement learning is used to fine-tune the cross-attention weights, dynamically explore the lesion correlation patterns between different modalities, and supplement the lesion features that cannot be captured by a single modality through modal complementarity, generating three sets of bimodal fusion feature vectors. The single-modal fusion feature vector and the dual-modal fusion feature vector are concatenated and input into the global attention layer. Based on the scene adaptation capability of meta-learning pre-training, the model dynamically allocates attention weights for different modalities and features according to the feature distribution of the input data. The feature adaptive correction module corrects the feature deviation in the fusion process in real time, and finally generates a global fusion feature vector with unified dimensions and containing multi-dimensional lesion information.

[0010] Furthermore, the lesion grading assessment stage constructs a standardized fish lesion grading system based on the global fusion feature vector output from the cross-modal fusion stage, using large-scale labeled data and semi-supervised comparative learning. The grading standard is formed by training and optimizing the labeled fish lesion data through semi-supervised comparative learning. Transfer learning enables the grading standard to quickly adapt to new fish species and new lesion types. Level 0 represents a healthy state with no lesion characteristics, all physiological and environmental parameters are within the appropriate range for the fish species, and there are no abnormal behavioral characteristics. Level 1 represents a mild lesion with no visible abnormalities on the body surface, only slight and recoverable fluctuations in physiological indicators, and no abnormalities in environmental parameters and behavioral characteristics, corresponding to the early incubation period of the disease. Grade 2 is mild lesion, characterized by visible lesions such as slight spots and congestion on the body surface, clear abnormalities in physiological indicators, and no obvious abnormalities in environmental parameters and behavioral characteristics, corresponding to the early stage of the disease; Grade 3 is moderate lesion, characterized by obvious ulceration, bleeding, fin damage, and other lesions on the body surface, significantly abnormal physiological indicators, slightly elevated environmental parameters, and mild behavioral abnormalities, corresponding to the middle stage of the disease; Grade 4 is severe lesion, characterized by severe ulceration, deformity, and large-area shedding on the body surface, severely abnormal physiological indicators, severely elevated environmental parameters, and significant behavioral abnormalities such as isolation, circling, and drifting, corresponding to the late stage of the disease, posing an extremely high risk of transmission. The multi-label joint learning model employs an improved CNN-LSTM and XGBoost fusion architecture, taking globally fused feature vectors as input and simultaneously completing three core tasks: 1) lesion level classification, outputting standardized grading results from 0 to 4, with a focus on improving the recognition accuracy of level 1 minor lesions; 2) lesion type identification, matching common fish lesion types such as gill rot, saprolegniasis, and ichthyophthirius multifiliis, supporting multi-label recognition of multiple lesions concurrently, and using an attention mechanism to enhance the discriminative ability of multiple lesion features; and 3) etiology tracing, outputting the inducing factors of lesions, including pathogen infection, environmental stress, and stress response, and combining Bayesian inference to improve the accuracy of etiology tracing. The model is trained using a semi-supervised contrastive learning approach, pre-training based on labeled lesion data and combining it with unlabeled aquaculture data, and then fine-tuning the model parameters using meta-learning.

[0011] The training process consists of three steps. First, the basic features of the model are learned using massive amounts of unlabeled aquaculture data to build a general feature recognition framework. Then, a small amount of labeled lesion data is used to fine-tune the model to enhance the ability to extract lesion features. Finally, the model output effect is continuously optimized by combining feedback data from actual aquaculture scenarios to ensure the model's recognition accuracy for different fish species and different lesion types, and to reduce the cost of large-scale manual labeling.

[0012] Furthermore, the intelligent triage decision-making stage constructs a reinforcement learning triage decision-making model based on the lesion level, lesion type, and etiology tracing results output by the lesion grading assessment stage. The triage decision-making model uses the lesion grading results, lesion type, etiology tracing results, real-time parameters of the aquaculture environment, fish species characteristics, and disease transmission risk as the state space, the isolation method, medication plan, water body control, disinfection measures, and monitoring frequency as the action space, and the reward function as the improvement of aquaculture survival rate, the reduction of disease spread risk, and the minimization of medication cost. It is pre-trained through scenario simulation and historical treatment data, and the transfer learning transfers the trained model parameters to the new aquaculture scenario. Based on the tiered assessment results, the triage decision model automatically outputs targeted triage and treatment instructions. Level 0 healthy individuals are subject to routine breeding management and continuous monitoring; Level 1 individuals with minor lesions are subject to individual isolation observation and environmental optimization; Level 2 individuals with mild lesions are subject to zoned isolation and targeted drug treatment; Level 3 individuals with moderate lesions are subject to independent isolation and intensive treatment and high-frequency effect evaluation; and Level 4 individuals with severe lesions are subject to harmless treatment and full-scene disinfection and control. During the triage and treatment process, data on the fish's condition, physiological indicators, and changes in environmental parameters after treatment are collected in real time and input into a reinforcement learning model. The model integrates a time-series prediction module to predict the treatment effect, and the model dynamically adjusts the treatment plan based on real-time feedback and prediction results.

[0013] Furthermore, the model iteration closed-loop stage, based on the treatment execution process and feedback data of the intelligent triage and decision-making stage, constructs a treatment effect feedback database. This database associates and stores data after triage and treatment, including treatment plans, changes in fish health, disease control effects, and changes in the aquaculture environment, with the corresponding original collected data and grading evaluation results, forming a dataset for model iteration. A fixed model iteration cycle is set, and based on the model iteration dataset, federated learning enables distributed training of data from multiple aquaculture bases. This jointly optimizes the feature extraction model, multimodal fusion model, lesion grading model, and triage and decision-making model, enabling the model to adapt to newly added lesion types, aquaculture scenarios, and fish species. A cloud-based integrated management platform is built to manage the entire process of data collection, preprocessing, feature extraction, fusion, hierarchical assessment, triage and treatment, and model optimization in a unified manner. The platform integrates a machine learning model monitoring module to monitor the model's running status in real time, automatically identify model drift and trigger iterative optimization, and also supports real-time data synchronization and command issuance to aquaculture management terminals and disease prevention and control platforms.

[0014] The beneficial effects of this invention are as follows: 1. This invention constructs a multi-source data acquisition terminal array and deploys it uniformly according to the aquaculture density. The terminal integrates an underwater high-definition binocular camera, visible and near-infrared hyperspectral imaging equipment, non-contact non-invasive physiological sensors, and multi-parameter water quality sensing units. It simultaneously collects three core modal data: visual, physiological, and environmental data, completes millisecond-level spatiotemporal alignment and metadata labeling, and performs modality adaptation preprocessing through meta-learning to construct a standardized, scene-adaptive multimodal dataset. It uses meta-learning-optimized algorithms such as isolated forest to complete abnormal data removal, feature enhancement, data repair, and standardization. It uses improved ResNet50, 3D-CNN, and bidirectional LSTM networks to extract features of each modality, and filters core lesion features through correlation analysis, redundant feature removal, and dynamic weight allocation. It uses a meta-learning-optimized cross-attention hierarchical fusion model to perform intra-modal fusion, cross-modal pairwise fusion, and global deep fusion, and completes feature correction to generate a global fusion feature vector containing multi-dimensional lesion information.

[0015] 2. This invention constructs a standardized fish disease grading system of 0-4 levels based on globally fused feature vectors, optimized by large-scale labeled data and semi-supervised comparative learning. It rapidly adapts to new fish species and novel disease types through transfer learning, and the grading standards align with industry norms for aquaculture disease prevention and control. A multi-label joint learning model combining an improved CNN-LSTM and XGBoost is employed, using globally fused feature vectors as input to simultaneously complete three tasks: disease level classification, disease type identification, and etiology tracing. Bayesian inference, attention mechanisms, and class balancing algorithms enhance the accuracy of multi-disease identification and etiology judgment. The model employs semi-supervised comparative learning pre-training and meta-learning for parameter fine-tuning, adapting to different fish species and aquaculture scenarios. Training is assisted by unlabeled aquaculture data, achieving integrated output of disease grading, type identification, and etiology tracing.

[0016] 3. Based on the lesion grading assessment results, this invention constructs a reinforcement learning triage decision model that integrates deep Q-networks and transfer learning. It establishes a state space containing lesion information, environmental parameters, and fish species characteristics, and an action space including isolation, medication, and water body control. A reward space is set based on aquaculture efficiency, automatically outputting targeted triage and treatment instructions. During the treatment process, real-time status and environmental feedback data are collected, and the treatment effect is predicted and the plan is dynamically adjusted using a time-series prediction module. An iterative dataset is constructed based on the feedback data from the entire triage and treatment process. Distributed model optimization across multiple aquaculture bases is achieved through federated learning. A cloud-based integrated management platform is built to uniformly manage the entire process data, integrating a model monitoring module to monitor operational status, identify model drift, and trigger iterative optimization. Attached Figure Description

[0017] Figure 1 This is a flowchart of the multimodal fish lesion grading assessment and triage process of the present invention; Figure 2 This is a flowchart illustrating the global fusion execution process of three types of modal features optimized by meta-learning in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figures 1 to 2 As shown, this embodiment of the invention provides a multimodal method for grading and triaging fish lesions, including the following specific steps: In this embodiment of the invention, a multi-source data acquisition terminal array is constructed during the data acquisition synchronization phase and uniformly deployed in the aquaculture water according to the stocking density. The terminal integrates an underwater high-definition binocular camera, visible and near-infrared hyperspectral imaging equipment, a non-contact non-invasive physiological sensing unit, and a multi-parameter water quality sensing unit to achieve synchronous acquisition of three core modal data: visual, physiological, and environmental. The visual modal data includes fish surface lesion characteristics and swimming behavior temporal characteristics. The physiological modal data is collected through non-contact photoplethysmography visual sensing and non-contact surface impedance non-invasive detection technology to collect physiological indicators such as fish respiratory rate, heart rate, and body surface temperature. The environmental modal data collects environmental parameters such as dissolved oxygen, pH value, and ammonia nitrogen in the water. The acquisition frequency of the three types of data is kept uniform. The deployment density of the multi-source data acquisition terminal array is matched according to the area of ​​the aquaculture water body and the amount of fish raised, with one acquisition terminal deployed for every 100 cubic meters of aquaculture water body. The terminal adopts a floating fixed installation method, with an installation depth between 0.5 meters and 1.5 meters underwater, which can be automatically adjusted according to the aquaculture water depth. The terminal has a built-in unified clock module with a clock synchronization accuracy of milliseconds, and the hardware acquisition frequency is uniformly set to 5 frames / second. The underwater high-definition binocular camera has a resolution of no less than 1920×1080, and the spectral range of the visible and near-infrared hyperspectral imaging equipment covers 400nm to 1700nm. The detection response time of the non-contact, non-invasive physiological sensing unit is less than 100 milliseconds. The multi-parameter water quality sensing unit can simultaneously collect five core environmental parameters: dissolved oxygen, pH value, ammonia nitrogen, nitrite, and water temperature.

[0020] The multi-parameter water quality sensing unit has a built-in automatic calibration module that performs zero-point calibration and range calibration daily using a standard solution, automatically eliminating detection deviations caused by long-term use. After calibration, the detection parameters are automatically updated. It also has an abnormal alarm function that immediately sends a reminder message to the cloud platform when the sensor malfunctions or calibration fails.

[0021] The non-contact, non-invasive physiological sensing method combines photoplethysmography visual sensing with non-invasive surface impedance detection. It captures changes in reflected light signals from the fish's body surface using an underwater camera, converts these changes into respiratory rate and heart rate data, and obtains the fish's body surface temperature parameters non-contactly through a surface impedance detection unit. The detection accuracy is sufficient to capture minute fluctuations in physiological indicators and identify physiological abnormalities caused by early lesions.

[0022] Using the unified clock of the acquisition terminal as a reference, the three types of modal data in the same time window and the same spatial area are spatiotemporally aligned at the millisecond level. Metadata tags are added to each set of aligned data. The tag content includes the farmed fish species, growth cycle, farming mode, farming density, and geographical area. The millisecond-level spatiotemporal alignment uses the built-in unified clock of the acquisition terminal as the sole reference, binding and aligning the three types of data—visual, physiological, and environmental—within the same 1-second time window and the same 5-square-meter spatial area, ensuring that the timestamp deviation of each set of data does not exceed 5 milliseconds and the spatial positioning deviation does not exceed 0.2 meters. After the system completes the spatiotemporal alignment, it automatically verifies the integrity and validity of the data, removes invalid combinations with out-of-tolerance timestamps, spatial positioning offsets, and missing data, and retains only multimodal data groups with consistent spatiotemporal matching. Metadata tags use a standardized encoding format. In addition to basic information, they also include auxiliary information such as data collection time, terminal number, water depth, weather conditions, and feeding status. The tags are stored in JSON format, and all tags are automatically generated by the system.

[0023] Meta-learning-based modal adaptation preprocessing enables the acquisition terminal to adaptively match the physiological characteristics differences of different fish species and the fluctuations of the breeding environment through a small number of pre-trained samples, thereby filtering out invalid data acquisition scenarios in advance and constructing a standardized, scenario-adaptive multimodal dataset.

[0024] The modality adaptation preprocessing is based on a model-independent meta-learning algorithm. It completes rapid pre-training with a small number of target fish species samples (5 to 10 groups), and the pre-training process only takes 10 minutes to complete feature space adaptation. The system has built-in automatic invalid data filtering rules, which can automatically identify invalid collection scenarios such as no fish targets, turbid water obstruction, equipment offline, and signal interruption, and directly discard useless data in that period, retaining only valid data that contains fish targets, has complete signals, and meets quality standards. The preprocessing process can adapt to differences in body size, body color, and swimming habits among different fish species, as well as changes in light, water temperature, and water turbidity in different aquaculture environments.

[0025] The meta-learning module automatically records the characteristic adaptation parameters of different fish species, forms exclusive adaptation templates locally, and stores them in a categorized manner. When encountering the same fish species or similar breeding environment again, the system directly calls the matching template to complete the rapid adaptation, reducing the debugging cost and time of on-site deployment. At the same time, the system continuously collects fish growth data and automatically updates the adaptation parameters based on changes in body shape, body color, and behavior during the fish's growth cycle.

[0026] In this embodiment of the invention, the data preprocessing stage is based on the standardized, scene-adaptive multimodal dataset constructed in the data acquisition and synchronization stage. For the three types of modal data, namely visual, physiological and environmental, the isolated forest algorithm based on meta-learning optimization is adopted. Through transfer learning, it adapts to the data fluctuation characteristics of different fish species and different aquaculture scenarios, identifies and removes outliers and outliers caused by sensor interference, water body obstruction and poor equipment contact, and improves the accuracy of outlier data identification. The meta-learning optimized isolated forest algorithm adopts an unsupervised anomaly detection architecture, which consists of three parts: a base forest construction module, a meta-feature adaptation module, and a domain adaptive transfer module. The base forest has 100 trees, 256 samples per tree, and a fixed anomaly score threshold of 0.7. It quickly locates anomaly samples through random subsampling and binary splitting. The meta-learning optimization module completes feature space adaptation through a small number of target fish species samples, automatically correcting the data distribution offset of different fish species and different aquaculture environments. The transfer learning uses the labeled conventional fish species dataset as the source domain and the newly added aquaculture species and unfamiliar aquaculture scenarios as the target domain. It achieves cross-scenario anomaly recognition through feature alignment and distribution calibration, and removes non-pathological anomalies such as water turbidity interference, equipment vibration, and signal packet loss.

[0027] The system marks abnormal data based on abnormal scores and performs an independent filtering process on the marked abnormal data. Abnormal data does not enter the subsequent feature extraction, model training and lesion determination stages, thereby reducing the impact of invalid data on model accuracy.

[0028] For visual modal data, an image denoising algorithm based on Vision Transformer is used to remove water body interference noise. An adaptive enhancement algorithm is used to highlight the features of lesion areas, and contrastive learning is used to optimize the feature enhancement effect, thereby strengthening the identification of weak features of early lesions. For physiological modal time-series data, an LSTM smoothing filtering algorithm is used to preserve the abnormal fluctuation trend of physiological indicators, and an attention mechanism is used to weight and highlight subtle fluctuation features related to lesions. For environmental modal time-series data, a Kalman filter algorithm is used to correct sensor drift errors and complete data benchmark calibration. A time-series prediction model is used to predict sensor drift trends in advance, thereby achieving early correction of sensor drift errors. The contrastive learning feature enhancement adopts a positive and negative sample comparison mechanism, using the features of the lesion area as positive samples and the normal body surface area and water background as negative samples. Through continuous iterative optimization, the model can more accurately distinguish between lesion features and interference information. For the weak spots, slight congestion and other inconspicuous features of early lesions, feature amplification and edge enhancement processing are used to make the weak features easier to be extracted and identified by the model.

[0029] To address short-term data gaps during data acquisition, a generative adversarial network is used to construct a time-series data repair model. Based on related modal data within the same time period, the model intelligently repairs missing single-modal data, preserving data integrity to the greatest extent possible. The min-max normalization algorithm and Z-score standardization algorithm are used to unify all data from the three modalities into the same dimensional range, eliminating the interference of different scales on the training of the machine learning model.

[0030] The visual modality denoising process employs the Vision Transformer image denoising algorithm to remove interference noise such as suspended matter in the water, light reflection, and water wave turbulence. The physiological modality time series data uses the LSTM smoothing filtering algorithm to remove random noise while preserving abnormal fluctuation characteristics, highlighting subtle changes in indicators such as respiration and heart rate. The environmental modality data uses the Kalman filtering algorithm to correct sensor drift errors in real time. For scenarios with short-term data loss, a generative adversarial network is used to construct a time series repair model, which fills in missing values ​​using data from the other two modalities within the same time period.

[0031] In this embodiment of the invention, the feature extraction and screening stage is based on the standardized data output from the data preprocessing stage. For visual modality image and video data, a pre-trained improved ResNet50 convolutional neural network is used to extract spatial features, and a 3D-CNN is used to extract behavioral temporal action features. These are fused to generate a visual modality feature matrix. The network parameters are fine-tuned through meta-learning, enabling the feature extraction model to quickly adapt to the differences in body surface features of newly added fish species. For physiological modality and environmental modality temporal data, a bidirectional LSTM network is used to extract dynamic features such as fluctuation trends and mutation amplitudes of the temporal data, generating physiological modality and environmental modality feature matrices respectively. The Transformer attention mechanism is combined to enhance the extraction effect of key features. The improved ResNet50 network, based on the standard residual structure, embeds SE channel attention modules in the 3rd and 4th residual blocks. It strengthens the feature weights of the lesion region through compression and activation operations. The network convolutional kernels are uniformly 3×3 in size, and the activation function is ReLU. After global average pooling, it outputs 2048-dimensional visual spatial features. The 3D-CNN network is used to extract the temporal features of fish swimming behavior. It consists of 3 three-dimensional convolutional layers and 2 three-dimensional max pooling layers. The temporal convolutional kernel size is 3×3×3 with a stride of 1. It captures dynamic features such as fish swimming speed, posture, and trajectory through spatiotemporal joint encoding. The bidirectional LSTM network used for physiological and environmental modalities has a hidden layer dimension of 256 and a time step of 16 frames. Combined with the Transformer multi-head attention mechanism, it enhances the extraction effect of temporal features that are highly correlated with lesions, such as fluctuations in physiological indicators and abrupt changes in environmental parameters. The features of the three modalities are independently encoded and then kept dimensionally aligned.

[0032] Based on the mutual information algorithm and Pearson correlation coefficient, the correlation between each extracted feature and the occurrence and development of fish lesions is calculated, and the contribution of each feature to lesion identification and classification is quantified. A correlation threshold is set, and redundant and noisy features with correlation below the threshold are removed. For the retained high-contribution features, the initial weights are dynamically assigned through reinforcement learning. The weights are adaptively adjusted according to the feature importance of different fish species and different lesion types to generate a filtered feature set of three modalities. While reducing the computational load of subsequent models, this method maximizes the retention of effective features related to lesions, especially weak features of early lesions, thereby improving the early identification capability of the model.

[0033] The feature correlation analysis employs a combination of mutual information algorithm and Pearson correlation coefficient. The threshold for the correlation between features and the occurrence and development of lesions is set to 0.3 by default. Redundant and noisy features below this threshold are directly removed to reduce the computational complexity of the model and the risk of overfitting. After dynamic weight allocation for high-contribution features through reinforcement learning, min-max normalization and Z-score standardization are used to achieve dimensionality unification. Min-max normalization linearly maps all feature data to the [0, 1] interval to eliminate dimensional differences. Z-score standardization uses a dataset mean of 0 and a variance of 1 to complete distribution calibration, ensuring that visual, physiological, and environmental modal features participate in subsequent fusion and classification within the same numerical space.

[0034] In this embodiment of the invention, the cross-modal fusion stage is based on the three types of modal-filtered feature sets generated in the feature extraction and filtering stage. It adopts a meta-learning optimized cross-attention hierarchical fusion model. The fusion model is divided into three fusion levels, which sequentially complete intramodal feature fusion, cross-modal pairwise feature fusion, and global cross-modal deep fusion. For the filtered feature sets of visual, physiological, and environmental modalities, a self-attention mechanism is adopted to complete the fusion of different features within the same modality, highlighting the core features with the highest correlation to the lesion within a single modality, and generating single-modal fusion feature vectors for the three modalities respectively. The meta-learning optimized cross-attention hierarchical fusion model consists of four core modules connected in series: a self-attention layer, a bidirectional cross-attention layer, a global attention layer, and a feature adaptive correction layer. Intramodal fusion employs a self-attention mechanism with eight attention heads to complete feature weighting and information condensation within a single modality, generating three types of single-modal fusion feature vectors. Cross-modal pairwise fusion uses a bidirectional cross-attention mechanism to achieve feature interaction and complementarity between vision and physiology, physiology and environment, and vision and environment, respectively. Through reinforcement learning, the cross-attention weights are dynamically fine-tuned to deeply explore the disease correlation patterns between modalities. Bidirectional cross-attention enables bidirectional information interaction between two types of modal features. It extracts the correlation features between a single modality and the other modality, and splices and integrates the correlation features generated by the interaction to enhance the expression effect of lesion-related information between modalities, so that the cross-modal fusion features can more comprehensively reflect the health status of fish. In the global fusion stage, the single-modal and dual-modal feature vectors are concatenated to a unified 1024-dimensional dimension. The weights are redistributed by the global attention layer, and the feature adaptive correction layer uses L2 regularization and residual connection to correct the fusion bias in real time. Finally, the output is a global fusion feature vector containing multi-dimensional lesion information, which retains the weak feature signals of early and mild lesions.

[0035] The correction module calculates the contribution deviation of each modality feature in real time. To address the problem that weak features of early lesions are easily masked by background information, it automatically increases the weight ratio of weak features of physiological and environmental modalities. At the same time, it suppresses the output influence of invalid features such as water interference and equipment noise, so that the global fusion features can completely retain the key information related to the lesions. Even in complex aquaculture scenarios such as insufficient light and turbid water, it can still maintain stable feature fusion accuracy.

[0036] A bidirectional cross-attention mechanism is adopted to complete the pairwise cross-modal feature fusion of vision and physiology, physiology and environment, and vision and environment. Reinforcement learning is used to fine-tune the cross-attention weights, dynamically explore the lesion correlation patterns between different modalities, and supplement the lesion features that cannot be captured by a single modality through modal complementarity, generating three sets of bimodal fusion feature vectors. The modal complementarity uses visual, physiological, and environmental data to corroborate and supplement each other. When there are no obvious pathological features in the visual modality, early lesions are identified through subtle fluctuations in the physiological modality. When physiological modality data is missing, the risk of lesions is judged through abnormal environmental parameters and behavioral characteristics. When environmental parameters are normal, lesions caused by non-environmental factors are ruled out through visual and physiological modalities. The three modalities complement each other to fill information gaps and avoid judgment errors caused by bias in data from a single modality.

[0037] The single-modal fusion feature vector and the dual-modal fusion feature vector are concatenated and input into the global attention layer. Based on the scene adaptation capability of meta-learning pre-training, the model dynamically allocates attention weights for different modalities and features according to the feature distribution of the input data. The feature adaptive correction module corrects the feature deviation in the fusion process in real time, and finally generates a global fusion feature vector with unified dimensions and containing multi-dimensional lesion information.

[0038] In this embodiment of the invention, the lesion grading assessment stage is based on the global fusion feature vector output by the cross-modal fusion stage, and constructs a standardized fish lesion grading system based on large-scale labeled data and semi-supervised comparative learning. The grading standard is formed by training and optimizing the labeled fish lesion data through semi-supervised comparative learning, which conforms to the industry norms for disease prevention and control in aquaculture. Transfer learning enables the grading standard to quickly adapt to new fish species and new lesion types. Level 0 is a healthy state with no lesion characteristics, all physiological and environmental parameters are within the appropriate range for the fish species, and there are no abnormal behavioral characteristics. Level 1 is a mild lesion with no visible abnormalities on the body surface, only slight and recoverable fluctuations in physiological indicators, and no abnormalities in environmental parameters and behavioral characteristics, corresponding to the early incubation period of the disease. The standardized lesion grading system is built on a massive amount of labeled data and uses a semi-supervised comparative learning method to complete the training and optimization. The training process is divided into three steps: pre-training with unlabeled data, fine-tuning with labeled data, and calibration in real-world scenarios. The grading standards are regularly updated in conjunction with aquaculture disease prevention and control guidelines, expert diagnostic experience, and actual treatment effects. For new fish species and new lesions, only a small number of labeled samples are needed to complete the adaptation of the grading standards through transfer learning.

[0039] The lesion grading system combines four aspects—physiological characteristics, environmental parameters, and behavioral characteristics—to make a comprehensive judgment. Different grades correspond to specific states of the four characteristics, and the system automatically matches the lesion grade based on the overall state of the four characteristics.

[0040] Grade 2 is mild lesion, characterized by visible lesions such as slight spots and congestion on the body surface, clear abnormalities in physiological indicators, and no obvious abnormalities in environmental parameters and behavioral characteristics, corresponding to the early stage of the disease; Grade 3 is moderate lesion, characterized by obvious ulceration, bleeding, fin damage, and other lesions on the body surface, significantly abnormal physiological indicators, slightly elevated environmental parameters, and mild behavioral abnormalities, corresponding to the middle stage of the disease; Grade 4 is severe lesion, characterized by severe ulceration, deformity, and large-area shedding on the body surface, severely abnormal physiological indicators, severely elevated environmental parameters, and significant behavioral abnormalities such as isolation, circling, and drifting, corresponding to the late stage of the disease, posing an extremely high risk of transmission. The multi-label joint learning model employs an improved CNN-LSTM and XGBoost fusion architecture, taking globally fused feature vectors as input and simultaneously completing three core tasks: 1) lesion level classification, outputting standardized grading results from 0 to 4, with a class balancing algorithm addressing the imbalance problem of samples at different lesion levels, focusing on improving the recognition accuracy of level 1 minor lesions; 2) lesion type identification, matching common fish lesion types such as gill rot, saprolegniasis, and ichthyophthirius multifiliis, supporting multi-label recognition of multiple lesions concurrently, and an attention mechanism enhancing the discriminative ability of multiple lesion features; 3) etiology tracing, outputting the inducing factors of lesions, including pathogen infection, environmental stress, and stress response, combined with Bayesian inference to improve the accuracy of etiology tracing. The model is trained using a semi-supervised contrastive learning approach, pre-training based on labeled lesion data and unlabeled aquaculture data, and fine-tuning model parameters through meta-learning to improve the model's generalization ability to different fish species and aquaculture scenarios.

[0041] The fusion architecture employs a serial connection method, inputting the spatial features extracted by the improved CNN into a bidirectional LSTM to complete temporal encoding. The encoding result is then input into an XGBoost multi-label classifier, simultaneously achieving three tasks: lesion grade classification, lesion type identification, and etiology tracing. Model training combines semi-supervised contrastive learning pre-training with meta-learning fine-tuning. The optimizer used is the Adam adaptive optimizer, with an initial learning rate set to 1e-4, a batch size of 32, and a total of 100 iterations. The learning rate is dynamically adjusted by decreasing by 10% every 20 iterations. The loss function uses multi-label cross-entropy loss, combined with a class balance factor to address the imbalance problem of samples from different lesion grades and types. Bayesian inference optimizes the etiology tracing output, focusing on improving the recognition accuracy of grade 1 minor lesions and concurrent lesion scenarios.

[0042] The multi-label recognition supports the simultaneous identification of two or more concurrent lesions. The system will output the results according to the severity of the lesions and the risk of transmission. It will match the corresponding feature differentiation rules for different lesion types and focus on the feature regions of different lesions through the attention mechanism to avoid mutual interference between the features of different lesions.

[0043] In this embodiment of the invention, the intelligent triage decision-making stage is based on the lesion level, lesion type, and etiology tracing results output by the lesion grading assessment stage. It constructs a reinforcement learning triage decision-making model based on the fusion of deep Q-network and transfer learning. The triage decision-making model uses the lesion grading results, lesion type, etiology tracing results, real-time parameters of the aquaculture environment, fish species characteristics, and disease transmission risk as the state space, the isolation method, medication plan, water body control, disinfection measures, and monitoring frequency as the action space, and the improvement of aquaculture survival rate, reduction of disease spread risk, and minimization of medication cost as the reward function. It is pre-trained through scenario simulation and historical treatment data. Transfer learning transfers the trained model parameters to new aquaculture scenarios, reducing model training costs and improving model adaptation efficiency. The deep Q-network triage decision-making model consists of an input layer, three fully connected hidden layers, an output layer, an experience replay module, and a target network module. The input layer has a 512-dimensional dimension and is used to receive state information such as lesion grading, environmental parameters, and fish species characteristics. The three hidden layers have dimensions of 256, 128, and 64 respectively, and the activation function is LeakyReLU. The output layer has a 32-dimensional dimension and corresponds to treatment actions such as isolation, medication, water body control, disinfection, and monitoring. The model's experience replay pool size is set to 10,000, the discount factor γ is 0.95, the learning rate is 5e-4, and the target network updates its parameters every 200 steps. The reward function adopts a multi-objective weighted formula: R = 0.5 × survival rate of aquaculture + 0.3 × (1 - risk of disease spread) - 0.2 × cost of medication. It minimizes the cost of medication while improving the survival rate and controlling the spread of disease. After the model is pre-trained with scenario simulation and historical treatment data, it can quickly adapt to new aquaculture scenarios and fish species through transfer learning.

[0044] The pre-trained model parameters are divided into a general decision layer and a scenario adaptation layer. The general decision layer retains the triage decision logic that is applicable across scenarios, while the scenario adaptation layer corresponds to different aquaculture conditions and fish species characteristics. When carrying out scenario migration, only the parameters of the scenario adaptation layer need to be updated, which can quickly adapt to different aquaculture modes such as factory farming, pond farming, and cage farming, as well as different aquaculture categories such as freshwater fish and marine fish.

[0045] Based on the tiered assessment results, the triage decision model automatically outputs targeted triage and treatment instructions. Level 0 healthy individuals are subject to routine breeding management and continuous monitoring; Level 1 individuals with mild lesions are subject to individual isolation and environmental optimization without drug treatment; Level 2 individuals with mild lesions are subject to zoned isolation and targeted drug treatment; Level 3 individuals with moderate lesions are subject to independent isolation and intensive treatment with high-frequency effect evaluation; and Level 4 individuals with severe lesions are subject to harmless treatment and full-scene disinfection and control to prevent the spread of disease. The harmless treatment is carried out in accordance with the standards for disease prevention and control in aquaculture. Severely diseased fish are collected in a sealed manner and disposed of harmlessly, and the process avoids the leakage of pathogens. The disinfection of the entire scene uses aquaculture-specific disinfectant to thoroughly disinfect the aquaculture water, aquaculture facilities, and fishing tools. After disinfection, the pathogen content in the water is continuously monitored until the standard is met before normal aquaculture management is restored.

[0046] During the triage and treatment process, data on the fish's condition, physiological indicators, and changes in environmental parameters after treatment are collected in real time and input into a reinforcement learning model. The model integrates a time-series prediction module to predict the treatment effect, and the model dynamically adjusts the treatment plan based on real-time feedback and prediction results.

[0047] The triage and treatment instructions are automatically sent from the cloud platform to the aquaculture site terminals. The execution process adopts a full recording mode, collecting treatment status data every 5 minutes, including fish activity status, physiological indicator recovery status, water parameter changes, medication implementation status, disinfection effect, etc. The feedback data is transmitted back to the reinforcement learning triage decision model in real time. The model, combined with the time series prediction module, predicts the treatment effect in advance from 1 hour to 24 hours. If the effect is found to be unsatisfactory or there is an abnormal risk, the treatment plan is automatically adjusted immediately, including adjusting the medication dosage, changing the isolation method, increasing the monitoring frequency, and strengthening water body control.

[0048] Based on the current state space parameters, the triage model selects the action with the best overall benefit from the action space and outputs standardized treatment instructions that include execution steps, execution duration, and monitoring frequency. The on-site terminals execute the instructions in an orderly manner.

[0049] In this embodiment of the invention, the model iteration closed-loop stage is based on the treatment execution process and feedback data of the intelligent triage and decision-making stage to construct a treatment effect feedback database. Data such as treatment plans, changes in fish health, disease control effects, and changes in the aquaculture environment after triage and treatment are associated and stored with the corresponding original collected data and graded evaluation results to form a dataset for model iteration. A fixed model iteration cycle is set, and based on the model iteration dataset, federated learning is used to achieve distributed training of data from multiple aquaculture bases. The feature extraction model, multimodal fusion model, lesion grading model, and triage and decision-making model are jointly optimized. Without the need for full retraining, the model can be adapted to newly added lesion types, aquaculture scenarios, and fish species. The federated learning adopts the FedAvg model average aggregation algorithm to complete distributed model iteration while ensuring the data privacy of each breeding base does not leave its local machine. The training rules are set to 5 rounds of local training per base, and the global model iteration cycle is 24 hours. Global model aggregation is performed once every 10 rounds of local training, and the aggregation weight is adaptively allocated according to the amount of data and model performance of each base. During the iteration process, the entire process of feature extraction, cross-modal fusion, lesion grading, and triage decision-making is optimized simultaneously. The cloud-integrated platform monitors model drift in real time and automatically completes parameter updates and multi-terminal synchronization after triggering iteration.

[0050] Each breeding base only uploads the gradient of the model parameters trained locally. The original breeding data is kept on the local terminal throughout the process and is neither transmitted to the outside nor centrally stored.

[0051] A cloud-based integrated management platform is built to manage the entire process of data collection, preprocessing, feature extraction, fusion, hierarchical assessment, triage and treatment, and model optimization in a unified manner. The platform integrates a machine learning model monitoring module to monitor the model's running status in real time, automatically identify model drift and trigger iterative optimization, and also supports real-time data synchronization and command issuance to aquaculture management terminals and disease prevention and control platforms.

[0052] The cloud-based integrated management platform is built with a distributed architecture and has six core functions: data storage, model management, real-time monitoring, command issuance, statistical analysis, and access control. All data in the process is stored in encrypted form and has a retention period of no less than 3 years. It can be retrieved, queried, and traced at any time. The data transmission adopts an encrypted transmission protocol, and is encrypted throughout the entire process from the data acquisition terminal to the cloud platform to prevent data leakage and tampering. The platform permission management is divided into three levels: administrator, technician and field operator. Different levels correspond to different data viewing, command operation and model adjustment permissions. The model monitoring module monitors key indicators such as feature extraction accuracy, grading accuracy, and triage execution effect in real time. When the indicators drop below the threshold, it automatically identifies model drift and immediately triggers the iterative optimization process. The platform supports multi-terminal synchronization on computers, mobile phones, and on-site terminals, and can connect to multiple breeding bases at the same time.

[0053] The model drift identification system has clear judgment criteria. When the accuracy of lesion grading decreases continuously, the triage and treatment effectiveness decreases, and the feature extraction accuracy is insufficient, the system determines that the model is drifting. After triggering the iteration, the system automatically retrieves the latest treatment feedback data, starts federated learning distributed training, and automatically verifies the model performance after training is completed. Once the model meets the criteria, the original model is replaced.

[0054] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal method for grading and triaging fish lesions, characterized in that, The specific steps include the following: During the data acquisition and synchronization phase, a multi-source data acquisition terminal array was constructed and uniformly deployed according to the breeding density. Three core modal data of visual, physiological and environmental data were collected synchronously. Millisecond-level spatiotemporal alignment and metadata labeling were completed. Based on meta-learning, modality adaptation preprocessing was performed on the collected data to construct a multimodal dataset. In the data preprocessing stage, based on the multimodal dataset, outlier removal, feature enhancement, data repair and standardization are completed to correct data bias and preserve data integrity, and output standardized data. In the feature extraction and screening stage, based on the preprocessed standardized data, features of each modality are extracted. Through correlation analysis and dynamic weight allocation, core features related to the lesion are screened to generate three types of modal feature sets after screening. In the cross-modal fusion stage, based on the selected three types of modal feature sets, the hierarchical fusion model sequentially completes feature fusion within the same modality, pairwise feature fusion across modalities, and global deep fusion. The model generates a global fusion feature vector containing multi-dimensional lesion information through dynamic weight allocation and feature adaptive correction. In the lesion grading assessment stage, a standardized lesion grading system of 0-4 levels is constructed based on global fusion feature vectors. Through a multi-label joint learning model, lesion level classification, lesion type identification and etiology tracing are completed simultaneously, and model pre-training and parameter fine-tuning are completed. In the intelligent triage decision-making stage, based on the results of the hierarchical assessment, a reinforcement learning triage decision-making model adapted to the fish farming scenario is constructed based on the fusion of deep Q network and transfer learning. The model constructs a state, action and reward space with multi-dimensional parameters including lesion grade, environmental parameters and fish species characteristics, outputs targeted triage and treatment instructions, and dynamically adjusts the treatment plan by combining real-time feedback and treatment effect prediction. In the closed-loop model iteration phase, an iterative dataset is constructed based on feedback data from the entire triage and treatment process. Distributed model iteration across multiple breeding bases is completed through federated learning, and an integrated cloud-based management platform is built.

2. The method for grading and triaging fish lesions based on multimodality according to claim 1, characterized in that, The multi-source data acquisition terminal array is uniformly deployed in the aquaculture water according to the aquaculture density. The terminal integrates an underwater high-definition binocular camera, visible and near-infrared hyperspectral imaging equipment, a non-contact non-invasive physiological sensing unit, and a multi-parameter water quality sensing unit. The synchronously acquired visual modal data includes the characteristics of fish skin lesions and the temporal characteristics of swimming behavior; the physiological modal data includes physiological indicators such as fish respiratory rate, heart rate, and body surface temperature; and the environmental modal data includes environmental parameters such as dissolved oxygen, pH value, and ammonia nitrogen in the water. Using the unified clock of the acquisition terminal as a reference, the three types of modal data in the same time window and the same spatial area are spatiotemporally aligned at the millisecond level. Metadata tags containing the aquaculture fish species, growth cycle, aquaculture mode, aquaculture density, and geographical area are added to each set of aligned data.

3. The method for grading and triaging fish lesions based on multimodality according to claim 2, characterized in that, The modality adaptation preprocessing is based on meta-learning. By pre-training with a small number of samples, the acquisition terminal can adaptively match the physiological characteristics differences of different fish species and the fluctuations of the breeding environment, filter out invalid data acquisition scenarios including those without disease signals and environmental interference in advance, and construct a standardized, scenario-adaptive multimodal dataset.

4. The method for grading and triaging fish lesions based on multimodality according to claim 3, characterized in that, The data preprocessing stage employs a meta-learning optimized isolated forest algorithm for the three types of modal data. Through transfer learning, it adapts to the data fluctuation characteristics of different fish species and aquaculture scenarios, identifying and removing outliers and isolated data. For visual modal data, an image denoising algorithm based on Vision Transformer is used to remove water body interference noise, and an adaptive enhancement algorithm is used to highlight the features of lesion areas. For physiological modal time series data, an LSTM smoothing filtering algorithm is used to preserve the abnormal fluctuation trends of physiological indicators. For environmental modal time-series data, a Kalman filter algorithm is used to correct sensor drift and complete benchmark calibration; for short-term data loss, a generative adversarial network is used to construct a time-series repair model to complete missing data repair based on associated modal data; and min-max normalization and Z-score normalization algorithms are used to unify the three types of modal data into the same dimensional range.

5. The method for grading and triaging fish lesions based on multimodality according to claim 4, characterized in that, The feature extraction and screening stage uses a pre-trained improved ResNet50 convolutional neural network to extract spatial features from visual modal image and video data, and a 3D-CNN to extract behavioral temporal action features. These features are then fused to generate a visual modal feature matrix. Meta-learning is used to fine-tune the network parameters to adapt to the differences in body surface features of newly added fish species. For time-series data of physiological and environmental modalities, a bidirectional LSTM network is used to extract time-series dynamic features, generating corresponding modal feature matrices. The Transformer attention mechanism is combined to enhance the extraction effect of key features. Based on the mutual information algorithm and Pearson correlation coefficient, the correlation degree between each feature and the occurrence and development of fish diseases is calculated and the contribution is quantified. A correlation degree threshold is set to remove redundant and noisy features with a correlation degree lower than the threshold. For the retained high contribution features, the weights are dynamically assigned through reinforcement learning. After adaptive adjustment, a filtered feature set of three modalities is generated.

6. The method for grading and triaging fish lesions based on multimodality according to claim 5, characterized in that, The cross-modal fusion stage adopts a meta-learning optimized cross-attention hierarchical fusion model, which completes the fusion of three levels in sequence. The first level targets the feature sets after filtering the three types of modalities, and uses a self-attention mechanism to complete the feature fusion within the same modality, generating three types of single-modal fusion feature vectors. The second layer employs a bidirectional cross-attention mechanism to complete the pairwise feature fusion of three sets of cross-modal features: visual and physiological, physiological and environmental, and visual and environmental. Through reinforcement learning, the cross-attention weights are fine-tuned to generate three sets of bimodal fusion feature vectors. The third layer concatenates the fusion feature vectors of the single-modality and dual-modality features and inputs them into the global attention layer. Attention weights are dynamically allocated according to the distribution of input features. The feature adaptive correction module corrects the fusion deviation in real time and finally generates a global fusion feature vector with uniform dimensions.

7. A multimodal fish lesion grading assessment and triage method according to claim 6, characterized in that, The standardized lesion grading system is a 0-4 level grading system, which is formed and optimized based on large-scale labeled data and semi-supervised comparative learning training. Through transfer learning, it can be adapted to new fish species and new lesion types. Level 0 represents a healthy state; Level 1 represents mild lesions during the early incubation period of the disease; Level 2 represents mild lesions during the initial stage of the disease; Level 3 represents moderate lesions during the middle stage of the disease; and Level 4 represents severe lesions during the late stage of the disease, posing an extremely high risk of transmission.

8. A multimodal fish lesion grading assessment and triage method according to claim 7, characterized in that, The multi-label joint learning model adopts an improved CNN-LSTM and XGBoost fusion architecture. It takes the global fusion feature vector as input and simultaneously completes three core tasks: first, lesion level classification, outputting standardized grading results from 0 to 4; second, lesion type identification, matching common fish lesion types including gill rot, saprolegniasis, and ichthyophthirius multifiliis, supporting multi-label identification of multiple lesions concurrently; and third, etiology tracing, outputting lesion inducing factors and combining Bayesian inference to improve the accuracy of etiology tracing. The model is pre-trained using semi-supervised contrastive learning and fine-tuned by meta-learning.

9. A multimodal fish lesion grading assessment and triage method according to claim 8, characterized in that, The reinforcement learning triage decision-making model is constructed based on the fusion of deep Q-networks and transfer learning. Its state space consists of lesion grading results, lesion type, etiology tracing results, real-time parameters of the aquaculture environment, fish species characteristics, and disease transmission risk. Its action space consists of isolation methods, medication regimens, water body control, disinfection measures, and monitoring frequency. The reward function is to improve survival rate, reduce disease spread risk, and minimize medication costs. The model is pre-trained using scenario simulations and historical treatment data, and adapted to new aquaculture scenarios through transfer learning. Based on the grading assessment results, the model outputs targeted triage and treatment instructions. During the treatment execution process, real-time feedback data is collected and input into the model, and the treatment plan is dynamically adjusted based on real-time feedback and treatment effect prediction.

10. A multimodal fish lesion grading assessment and triage method according to claim 9, characterized in that, The model iteration closed-loop stage associates and stores triage and treatment plans, changes in fish health, disease control effects, and changes in the aquaculture environment with the corresponding original collected data and graded assessment results, constructing a treatment effect feedback database and a model iteration dataset. A fixed model iteration cycle is set, and distributed training across multiple aquaculture bases is achieved through federated learning based on the iteration dataset. The entire process model is jointly optimized to adapt to new disease types, aquaculture scenarios, and fish species. The cloud-based integrated management platform provides unified management of the entire process data, integrates a model monitoring module, monitors the model's operating status in real time, automatically identifies model drift and triggers iterative optimization, and supports real-time data synchronization and command issuance across multiple terminals.

Images