Fish tank water quality monitoring method and system based on multimodal fusion

Abstract

The present invention provides a fish tank water quality monitoring method and system based on multimodal fusion, which is applied to the field of fish tank water quality monitoring. The method includes: obtaining a fish tank water image; inputting the preprocessed fish tank water image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes: performing water color analysis on the water color image to obtain a water color feature vector; performing fish behavior analysis on a fish behavior video to obtain a fish behavior feature vector; performing water physical phenomenon analysis on the water physical phenomenon image to obtain a water physical phenomenon feature vector; and performing multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image. The present invention can provide a more accurate and predictive water quality health status score.

Description

Technical Field

[0001] The present invention relates to the technical field of fish tank water quality monitoring, and in particular to a fish tank water quality monitoring method and system based on multimodal fusion. Background Art

[0002] Currently, monitoring water quality in home aquariums relies primarily on manual observation and specialized water quality monitoring equipment. For ornamental fish raised at home, water quality is often assessed by observing characteristics such as water color and odor, or by using pH and ammonia nitrogen monitoring equipment.

[0003] However, these traditional methods have many limitations. For example, human visual observation is easily affected by subjective factors, manual monitoring is prone to miss sudden changes in water quality, and timely warning and treatment are difficult, and water quality indicators cannot be accurately assessed. Traditional water quality monitoring relies on electrochemical sensors (such as pH meters and dissolved oxygen probes), which have problems such as electrode aging, frequent calibration, and inability to detect microbial contamination. A single modality is difficult to reflect the comprehensive status of water quality.

[0004] It can be seen that the fish tank water quality monitoring method in the related art has the technical problem of being difficult to reflect the comprehensive status of water quality. Summary of the Invention

[0005] The present invention provides a fish tank water quality monitoring method and system based on multimodal fusion, which is used to solve the defect that the fish tank water quality monitoring method in the prior art is difficult to reflect the comprehensive status of water quality, and to provide a more accurate and predictive water quality health status score.

[0006] The present invention provides a fish tank water quality monitoring method based on multimodal fusion, comprising the following steps: obtaining a fish tank water image, wherein the fish tank water image comprises: a water color image, a fish behavior video, and a water physical phenomenon image; inputting the preprocessed fish tank water image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, wherein the method comprises: performing water color analysis on the water color image to obtain a water color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water physical phenomenon analysis on the water physical phenomenon image to obtain a water physical phenomenon feature vector; and performing multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image.

[0007] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, water color analysis is performed on the water color image to obtain a water color feature vector, including: constructing a spatiotemporal feature extraction network with a general video understanding model as the backbone network; pre-training the spatiotemporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatiotemporal feature extraction network; inserting a trainable parameter adapter module into the pre-trained spatiotemporal feature extraction network for domain fine-tuning to obtain a fine-tuned spatiotemporal feature extraction network; using a Focal Loss loss function to optimize the classification task of the fine-tuned spatiotemporal feature extraction network, and introducing a chlorophyll concentration regression task to obtain a trained spatiotemporal feature extraction network; inputting the water color image into the trained spatiotemporal feature extraction network to obtain a water color feature vector output by the trained spatiotemporal feature extraction network.

[0008] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, fish behavior analysis is performed on the fish behavior video to obtain a fish behavior feature vector, including: extracting features from the fish behavior video through a multi-scale target detection network based on an attention mechanism to obtain a multi-level feature map; extracting features from the multi-level feature map through a deep convolutional neural network to obtain fish body morphological features; migrating the knowledge of a general video understanding model to the field of fish behavior analysis, and fine-tuning it to obtain a fish behavior recognition model; and identifying the fish body morphological features through the fish behavior recognition model to obtain a fish behavior feature vector.

[0009] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, the water physical phenomenon image is subjected to water physical phenomenon analysis to obtain a water physical phenomenon feature vector, including: extracting features of floating objects, abnormal bubbles and suspended particles on the water surface of the water physical phenomenon image based on a multi-scale attention mechanism through an image understanding model to obtain a water physical phenomenon feature vector.

[0010] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, multimodal fusion analysis is performed based on the water color feature vector, the fish behavior feature vector and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image, including: determining the environmental adaptability weight according to the static weight preset based on expert knowledge and the dynamic weight learned based on historical data; performing multimodal complementary enhancement based on the water color feature vector, the fish behavior feature vector and the water physical phenomenon feature vector to obtain an enhanced feature vector of each modality, wherein the modalities include: water color modality, fish behavior modality and water physical phenomenon modality; performing temporal attention fusion based on the water quality characteristics at the target time point and the water quality characteristics at the historical time point to obtain the temporal attention fusion feature of the target time point; determining the comprehensive water quality score corresponding to the fish tank water image based on the environmental adaptability weight, the enhanced feature vector of each modality and the temporal attention fusion feature of the target time point.

[0011] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, the comprehensive water quality score corresponding to the fish tank water image is determined based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature of the target time point, including:

[0012]

[0013] in, Indicates the comprehensive water quality score. represents the normalization function, represents the modal index, represents the temporal attention fusion feature of the target time point, Indicates the The enhanced eigenvectors of the modes, Represents Different modal indexes, Indicates the The mode and The environmental adaptability weight between the modalities, Indicates the The enhanced eigenvectors of the modes.

[0014] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, after performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image, the method also includes: obtaining a user query instruction; mapping the user query instruction to a predefined intention category set through a pre-trained language model to obtain a user intention category; performing vector similarity search in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector and the water body physical phenomenon feature vector to obtain a target historical case; according to a cross-modal attention mechanism, fusing the user intention category with the target historical case to obtain a fused feature vector; performing vector search in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector, the water body physical phenomenon feature vector and the target historical case to obtain a knowledge base retrieval result; and generating an inference answer result corresponding to the user query instruction based on the fused feature vector and the knowledge base retrieval result through retrieval enhancement generation.

[0015] The present invention also provides a fish tank water quality monitoring system based on multimodal fusion, comprising the following modules: an acquisition module, for acquiring a fish tank water body image, wherein the fish tank water body image comprises: a water body color image, a fish behavior video, and a water body physical phenomenon image; a water quality assessment module, for inputting the preprocessed fish tank water body image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, wherein the module comprises: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; and performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-described methods for monitoring fish tank water quality based on multimodal fusion.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for monitoring water quality in a fish tank based on multimodal fusion as described above is implemented.

[0018] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for monitoring fish tank water quality based on multimodal fusion.

[0019] The fish tank water quality monitoring method and system based on multimodal fusion provided by the present invention comprehensively cover multiple key influencing factors of fish tank water quality (such as chemical indicators, biological behavior, and physical state) by simultaneously acquiring water color images, fish behavior videos, and water physical phenomenon images; feature vectors are extracted from water color, fish behavior, and physical phenomena respectively, avoiding the limitations of a single data source and being able to more comprehensively reflect the water quality status; by analyzing water color images, water quality anomalies can be quickly identified; by analyzing fish behavior videos, the health status of the water body can be indirectly assessed; by analyzing water physical phenomenon images (such as bubbles, turbidity, etc.), the cause of water quality anomalies can be further verified; combining the above three types of modalities, different data sources can be cross-validated to reduce misjudgment and improve the accuracy and reliability of the comprehensive water quality score; through the comprehensive water quality score, users can intuitively understand the current water quality status and promptly discover potential problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flow chart of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention.

[0022] Figure 2 It is a schematic diagram of the overall process of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention.

[0023] Figure 3 It is a schematic diagram of the hardware implementation device of the multimodal fusion fish tank water quality monitoring method provided by the present invention.

[0024] Figure 4 It is a module schematic diagram of the fish tank water quality monitoring system based on multimodal fusion provided by the present invention.

[0025] Figure 5 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] Home fish farming is growing in popularity worldwide and has become a popular hobby. However, due to a lack of professional knowledge and experience, many families encounter numerous problems in fish management, such as improper water quality monitoring, inappropriate feeding practices, and untimely disease prevention and treatment. These problems can lead to damage to the health of their fish and even death. Aquarium water quality is a key factor affecting fish health. Deteriorating water quality can lead to a range of problems, including respiratory distress, decreased immunity, and increased disease. Therefore, effective monitoring and management of aquarium water quality is crucial to maintaining the health of ornamental fish.

[0028] Currently, monitoring water quality in home aquariums relies primarily on manual observation and specialized water quality monitoring equipment. For ornamental fish kept at home, water quality is typically assessed by observing characteristics such as water color and odor, or by using pH and ammonia nitrogen monitoring equipment. These traditional methods have numerous limitations. First, human observation is susceptible to subjective factors, and manual monitoring can easily miss sudden changes in water quality, making it difficult to provide timely warnings and interventions, and unable to accurately assess water quality indicators. Second, traditional water quality monitoring relies on electrochemical sensors (such as pH meters and dissolved oxygen probes), which are subject to issues such as electrode aging, frequent calibration requirements, and an inability to detect microbial contamination. A single modality fails to comprehensively reflect water quality. Furthermore, specialized water quality testing equipment is expensive, making it unsuitable for the average household. These issues lead to suboptimal performance in home aquarium fish farming, difficulties in ensuring fish health, and high aquaculture costs.

[0029] In response to the above problems, it is necessary to develop a low-cost, easy-to-operate, intelligent fish tank water quality monitoring system to meet the needs of the majority of home fish-raising enthusiasts. Based on artificial intelligence technology, the present invention proposes a fish tank water quality monitoring method, system and device based on multimodal fusion. By using machine vision and deep learning algorithms, through the analysis of multiple modal combinations of fish tank images and videos, automatic evaluation and real-time early warning of water quality indicators are achieved, greatly improving monitoring efficiency and accuracy. At the same time, by continuously monitoring changes in fish behavior and appearance, signs of fish disease can be detected early, helping users to take preventive measures in a timely manner. The present invention can significantly reduce the risk of fish diseases, extend the use cycle of fish tanks, save maintenance costs, and thus improve the success rate and user experience of home ornamental fish breeding.

[0030] The present invention proposes a fish tank water quality monitoring method, system and device based on multimodal fusion, aiming to solve the problems of traditional home ornamental fish water quality monitoring, such as reliance on subjective manual observation, incomplete and high-cost monitoring by single-modal sensors, and difficulty in providing real-time warnings of water quality deterioration and fish health risks.

[0031] The present invention collects fish tank images and video data through high-definition cameras, and conducts deep learning analysis on three modalities: water color, fish behavior, and water physical phenomena. At the same time, it adopts three mechanisms: environmental adaptability weight adjustment, multimodal complementary enhancement, and temporal attention fusion to fuse multimodal data. Finally, it makes early warning judgments based on the water quality health score calculated based on the fusion results.

[0032] In addition, it is equipped with a knowledge base and natural language processing modules, which support users to understand water quality conditions and treatment suggestions through question-and-answer interactions. The hardware uses a suction cup high-definition camera and a low-power data processor, which is low-cost and easy to install. The present invention introduces a multimodal fusion interaction mechanism into the field of home fish tank monitoring, replacing traditional sensors with visual analysis to reduce hardware costs, significantly improve monitoring sensitivity and predictive capabilities, and integrate lightweight hardware with a knowledge base-driven interactive system to achieve low-power, highly usable intelligent management. The present invention can significantly improve the efficiency and success rate of home ornamental fish breeding, and reduce the risk of fish diseases and maintenance costs.

[0033] Optionally, the fish tank water quality monitoring method based on multimodal fusion in the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the fish tank water quality monitoring method based on multimodal fusion in this embodiment executed by a server as an example.

[0034] Figure 1 This is a flow chart of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0035] Step 101: Acquire a fish tank water image, wherein the fish tank water image includes: a water color image, a fish behavior video, and a water physical phenomenon image.

[0036] In an embodiment of the present invention, to obtain high-quality images of the fish tank water, the present invention uses a high-definition camera to capture the fish tank in real time. The high-definition camera has excellent imaging performance and can output high-resolution, high-dynamic range images and videos.

[0037] The HD camera supports flexible capture modes, including timed and continuous capture, tailored to monitoring needs. Timed capture automatically captures images of the aquarium water at set intervals, while continuous capture continuously records over a specific period, capturing dynamic changes in water quality. This allows for the acquisition of sufficient image samples without disrupting the normal operation of the aquarium, providing a reliable data foundation for subsequent water quality analysis.

[0038] Step 102: input the pre-processed fish tank water image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model.

[0039] In the embodiment of the present invention, necessary preprocessing is performed on the collected fish tank water image, such as size normalization, pixel normalization, etc., to meet the format requirements of the fish tank water quality assessment model input.

[0040] The preprocessed fish tank water image is then fed into a water quality assessment model, where it undergoes a series of operations, such as convolution and pooling, to extract high-level semantic features. The model ultimately outputs a confidence score (comprehensive water quality score) for each water quality category, reflecting its understanding and judgment of the image content.

[0041] To train a fish tank water quality assessment model, it is necessary to collect a large dataset of fish tank water images. This dataset should be divided into a training set and a test set. The model should be trained on the training set, and the model's hyperparameters should be adjusted using methods such as cross-validation and grid search to obtain optimal weights and optimize model performance. The fish tank water image dataset should cover a variety of common water quality conditions, such as clear, turbid, green, and yellow water, as well as various fish activity characteristics. Each fish tank water image sample should be annotated to indicate its true water quality category. To improve model robustness, the fish tank water image dataset should also include images under varying lighting, viewing angle, and other factors. To enhance model generalization, data augmentation techniques such as random cropping, rotation, and flipping can be used to increase sample diversity. Sufficient data volume and reliable annotation quality are key to training a high-performance model.

[0042] Wherein, step 102 specifically includes the following steps:

[0043] Step 1021: Perform water color analysis on the water color image to obtain a water color feature vector.

[0044] In some embodiments, a convolutional neural network (CNN) or a Vision Transformer (such as SwinTransformer) is used as the backbone network. These models can effectively extract local and global color features of images.

[0045] The preprocessed water color image is fed into the model, where color features are extracted through multi-layer convolution or self-attention mechanisms. The final layer of the model (typically a fully connected layer or global pooling layer) outputs a water color feature vector.

[0046] Step 1022: Perform fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector.

[0047] In some embodiments, fish behavior videos are preprocessed and the resulting video frame sequence is fed into a fish behavior analysis model. The model uses spatiotemporal convolution or self-attention mechanisms to extract spatial features (e.g., fish posture and position) and temporal features (e.g., swimming trajectories and behavioral dynamics) from the video frames layer by layer. Ultimately, the model outputs a fixed-length feature vector representing the fish's behavioral information.

[0048] Step 1023: Perform water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector.

[0049] Use a convolutional neural network (CNN) or a Vision Transformer (such as the Swin Transformer) as the backbone network. These models effectively extract both local and global features of images. Preprocessed images of water physics phenomena are fed into the model, where physical phenomena features are extracted through multi-layer convolution or self-attention mechanisms. The final layer of the model (typically a fully connected layer or global pooling layer) outputs a water physics feature vector.

[0050] Step 1024 , performing multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector, to obtain a comprehensive water quality score corresponding to the fish tank water image.

[0051] refer to Figure 2 , Figure 2 This is a schematic diagram of the overall process of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention, which includes: image acquisition to obtain water color images, fish behavior videos and water physical phenomenon images, image preprocessing, training data sets, model training, obtaining optimal weights, water quality analysis, water color analysis and fine-tuning to obtain water color feature vectors, fish behavior analysis and fine-tuning to obtain fish behavior feature vectors, water physical phenomenon analysis and fine-tuning to obtain water physical phenomenon feature vectors, multimodal fusion analysis, question and answer interaction, single-modal question and answer, and early warning reminders.

[0052] In this embodiment of the present invention, the water quality analysis module is divided into three parts: water color analysis, fish behavior analysis, and water physical phenomenon analysis. These modules provide comprehensive, intelligent analysis and assessment of aquarium water quality. This comprehensive analysis of multi-dimensional data effectively improves the accuracy, timeliness, and comprehensiveness of water quality monitoring.

[0053] Through the above steps of the embodiment of the present invention, by simultaneously acquiring water color images, fish behavior videos and water physical phenomenon images, multiple key influencing factors of fish tank water quality (such as chemical indicators, biological behavior, physical state) are fully covered; feature vectors are extracted from water color, fish behavior and physical phenomena respectively, avoiding the limitations of a single data source and being able to more comprehensively reflect the water quality status; by analyzing water color images, water quality anomalies can be quickly identified; by analyzing fish behavior videos, the health status of the water body can be indirectly evaluated; by analyzing water physical phenomenon images (such as bubbles, turbidity, etc.), the cause of water quality anomalies can be further verified; combining the above three types of modalities, different data sources can be cross-validated to reduce misjudgment and improve the accuracy and reliability of the comprehensive water quality score; through the comprehensive water quality score, users can intuitively understand the current water quality status and discover potential problems in a timely manner.

[0054] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, water color analysis is performed on a water color image to obtain a water color feature vector, including:

[0055] Build a spatiotemporal feature extraction network with a general video understanding model as the backbone network;

[0056] Pre-training the spatiotemporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatiotemporal feature extraction network;

[0057] Insert the trainable parameter adapter module into the pre-trained spatiotemporal feature extraction network for domain fine-tuning to obtain the fine-tuned spatiotemporal feature extraction network;

[0058] The Focal Loss function is used to optimize the classification task of the fine-tuned spatiotemporal feature extraction network, and the chlorophyll concentration regression task is introduced to obtain the trained spatiotemporal feature extraction network;

[0059] The water body color image is input into the trained spatiotemporal feature extraction network to obtain the water body color feature vector output by the trained spatiotemporal feature extraction network.

[0060] In an embodiment of the present invention, the spatiotemporal feature extraction capability of general video understanding models (such as Video Swin Transformer, TimeSformer, etc.) is migrated to the field of water color analysis. Through multimodal data fusion and domain adaptive fine-tuning, a dedicated model is constructed that can accurately identify six types of water colors (clear and transparent, light green, brown / yellow, turbid white, blue-green, and milky white / gray).

[0061] For example, using the Video Swin Transformer as the backbone network, a spatiotemporal feature extraction network was constructed to capture the dynamic characteristics of water bodies. A two-stage transfer learning strategy was implemented, first pre-training basic features on ImageNet-21K and aquatic video datasets. A trainable Adapter module was then inserted for domain fine-tuning. Focal Loss was used to address sample imbalance, and a chlorophyll concentration regression task was introduced to enhance model interpretability, resulting in a trained spatiotemporal feature extraction network. The trained spatiotemporal feature extraction network combines visual features with water quality parameters to output color categories and corresponding ecological causes, providing an important basis for subsequent water quality warnings.

[0062] The trained spatiotemporal feature extraction network identifies clear water bodies through their unique light refraction patterns and visibility metrics. Clear water bodies generally indicate a healthy and balanced aquatic environment, including appropriate dissolved oxygen levels, minimal suspended solids, and controlled nutrient levels.

[0063] When the trained spatiotemporal feature extraction network detects a greenish hue in the water, it triggers analysis of algal growth indicators. Greenish hues are often caused by eutrophication due to excess nutrients (phosphorus and nitrogen) combined with sufficient light penetration.

[0064] The trained spatiotemporal feature extraction network identifies brown or yellow water as an indicator of elevated tannin levels or decomposition of organic matter. Brown or yellow water is often caused by decomposing humus (such as drifting wood) or excessive fish food in the water.

[0065] The trained spatiotemporal feature extraction network identifies white turbid water bodies through the reflectance and scattering patterns characteristic of high bacterial populations or suspended organic particles. By analyzing turbidity levels and patterns, bacterial concentration levels and the severity of organic loads are estimated, alerting users to intervene before critical thresholds are reached.

[0066] When the trained spatiotemporal feature extraction network identifies blue-green water, this typically indicates a cyanobacterial bloom. Due to potential toxin production, this situation requires immediate user attention. The bloom's stage of development can be assessed by analyzing color intensity, distribution patterns, and temporal progression.

[0067] When a trained spatiotemporal feature extraction network identifies milky or gray water bodies—colors often caused by undissolved chemicals, including fertilizers, pH adjusters, or water treatment compounds—color distribution, opacity levels, and sedimentation characteristics are evaluated to distinguish temporary treatment effects from potentially harmful chemical imbalances.

[0068] Through the embodiments of the present invention, a spatiotemporal feature extraction network based on a general video understanding model as its backbone can effectively capture the spatiotemporal characteristics of water color images, including color distribution and texture variations. By pre-training on a water video dataset, the model can learn general features related to water bodies. By inserting a trainable parameter adapter module for domain fine-tuning, the model can quickly adapt to the specific scenarios of fish tank water bodies, improving feature extraction accuracy. Using the FocalLoss loss function to optimize classification tasks can effectively address sample imbalance (e.g., a small number of samples in certain color categories) and enhance the model's ability to recognize rare categories. By introducing a chlorophyll concentration regression task, the model not only extracts color features but also learns implicit information related to chlorophyll concentration, enhancing the model's robustness and interpretability.

[0069] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, fish behavior analysis is performed on fish behavior videos to obtain fish behavior feature vectors, including:

[0070] The fish behavior video is subjected to feature extraction through a multi-scale object detection network based on the attention mechanism, and a multi-level feature map is obtained.

[0071] The multi-level feature map is extracted through a deep convolutional neural network to obtain the morphological characteristics of the fish body;

[0072] Transfer the knowledge of the general video understanding model to the field of fish behavior analysis and fine-tune it to obtain a fish behavior recognition model;

[0073] The fish behavior recognition model is used to identify the fish morphological characteristics and obtain the fish behavior feature vector.

[0074] In an embodiment of the present invention, the fish behavior analysis module includes: target detection and individual identification, feature extraction and behavior recognition.

[0075] For target detection, a multi-scale target detection network based on an attention mechanism is employed. This network first extracts multi-layer feature maps using a backbone network (such as a modified ResNet or EfficientNet). It then introduces customized spatial attention and channel attention modules. The spatial attention module dynamically learns the spatial distribution characteristics of fish targets in the image, effectively suppressing background noise. The channel attention module adaptively adjusts the importance of different feature channels, enhancing the network's ability to perceive subtle morphological features of fish, thereby enabling precise localization and identification of individual fish underwater. This model, through its multi-scale feature extraction technology, effectively handles the complexities of fish size, posture, and background in underwater environments.

[0076] Deep convolutional neural networks (such as DenseNet or ResNeXt variants) extract fish morphological features, including surface texture, fin posture, and gill status. Combining 3D convolution with a temporal attention mechanism captures the dynamics of fish movement patterns. Physiological state feature modeling utilizes fine-grained regional analysis, focusing on health indicators such as breathing, swimming, and body surface condition. Self-supervised representation learning reduces reliance on large amounts of labeled data and improves the model's adaptability to new environments.

[0077] The knowledge of the general video understanding model was transferred to the field of fish behavior analysis. The general video understanding model was fine-tuned to obtain a large model for fish behavior recognition. The morphological characteristics of the fish body were identified, and the following fish behaviors were obtained.

[0078] Fish exhibit rapid breathing: Key symptoms include rapid opening and closing of the gill covers, frequent surfacing, and unusual swimming patterns. By analyzing deviations from normal breathing rates, the fish behavior recognition model determined that this behavior is highly correlated with insufficient dissolved oxygen in the water.

[0079] Fin folding and sluggish swimming: By precisely tracking the fish's movement and body posture, subtle changes in fin folding and sluggish swimming can be detected. These behavioral patterns are typically manifested as a significant decrease in swimming speed, slower turning movements, and a decrease in fin spread angle. By integrating this behavioral data with water quality parameters, particularly ammonia nitrogen and nitrite levels, the fish behavior recognition model can establish a correlation between behavioral changes and excessive toxins and assess the severity of water quality issues based on the degree of behavioral abnormality. For example, the fish behavior recognition model may determine that the current situation is due to excessive toxins (such as ammonia or nitrite) in the water.

[0080] Fish with spots or increased mucus: Specifically trained to detect changes in fish surface conditions, the system can detect signs such as spots, abnormally increased mucus secretion, and color changes. High-precision image analysis and time series comparisons can distinguish between normal surface changes and abnormalities caused by pathogenic infection. The fish behavior recognition model can determine the correlation between such behavior and deteriorating water quality, such as a possible bacterial infection.

[0081] Through the embodiments of the present invention, a multi-scale target detection network based on the attention mechanism can simultaneously capture local details (such as fish posture and movements) and global context (such as swimming trajectories and group behavior) in fish behavior videos. The attention mechanism can dynamically focus on key areas in the video (such as fish position and behavior changes) to improve the accuracy of feature extraction. Using a deep convolutional neural network to further extract features from multi-level feature maps can effectively capture the morphological characteristics of the fish (such as fish outline, posture, and movement). The knowledge of general video understanding models (such as Video Swin Transformer or I3D) is transferred to the field of fish behavior analysis, and its powerful spatiotemporal feature extraction capabilities are utilized to quickly adapt to new tasks. By fine-tuning on the fish behavior dataset, the model can learn specific features related to fish behavior and improve the accuracy and robustness of behavior recognition.

[0082] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, water physical phenomenon analysis is performed on a water physical phenomenon image to obtain a water physical phenomenon feature vector, including:

[0083] The image understanding model is used to extract features of floating objects, abnormal bubbles and suspended particles from images of water physical phenomena based on a multi-scale attention mechanism to obtain the feature vector of the water physical phenomenon.

[0084] In this embodiment, based on image understanding models (such as CLIP and DINOv2), the system achieves end-to-end intelligent diagnosis of floating objects, abnormal bubbles, and suspended matter by decoupling fine-grained visual features and reasoning about physical properties. For example, with ViT-Hybrid (a fusion of CNN and Transformer) at its core, a multi-scale attention mechanism is used to extract the morphological and dynamic characteristics of floating objects (fish food residues, fecal clumps), abnormal bubbles (diameter > 2mm and residence time > 10 seconds), and suspended particles (particle size > 50μm).

[0085] Floating objects or sediments are fish food residue or excrement, which can deteriorate water quality. Bubble retention: Abnormal bubbles with a diameter greater than 2mm and a retention time greater than 10 seconds may indicate abnormal surface tension in the water, often associated with excessive organic matter. Suspended matter in the water: Particles with a diameter greater than 50μm indicate high turbidity and the filtration system may need cleaning.

[0086] Through the embodiments of this invention, the local details and global distribution of water physical phenomena are captured through an attention mechanism and multi-scale feature extraction. Floating objects, abnormal bubbles, and suspended particles are accurately detected, and their key features are extracted. The resulting feature vectors of water physical phenomena comprehensively characterize the physical state of the water, providing important input for subsequent water quality assessment.

[0087] According to the present invention, a method for monitoring fish tank water quality based on multimodal fusion is provided. Multimodal fusion analysis is performed based on water color feature vectors, fish behavior feature vectors, and water physical phenomenon feature vectors to obtain a comprehensive water quality score corresponding to the fish tank water image, including:

[0088] Determine the environmental adaptability weight based on the static weight preset based on expert knowledge and the dynamic weight learned based on historical data;

[0089] Performing multimodal complementary enhancement based on water color feature vectors, fish behavior feature vectors, and water physical phenomenon feature vectors to obtain an enhanced feature vector for each mode, wherein the modes include: water color mode, fish behavior mode, and water physical phenomenon mode;

[0090] Based on the water quality characteristics of the target time point and the water quality characteristics of the historical time points, the temporal attention fusion feature of the target time point is obtained;

[0091] The comprehensive water quality score corresponding to the fish tank water image is determined based on the environmental adaptability weight, the enhanced feature vector of each modality and the temporal attention fusion features of the target time point.

[0092] In an embodiment of the present invention, by introducing an environmental adaptability weight adjustment mechanism, the multimodal fish tank water quality monitoring system can intelligently balance prior knowledge and historical data, rely more on expert experience when the environment is stable, and pay more attention to data learning results when the environment changes, thereby overcoming the limitations of the traditional fixed weight system, improving the system's adaptability, generalization ability and reliability, achieving adaptive decision-making, reducing manual intervention, and providing more intelligent technical support for fish tank water quality monitoring.

[0093] The environmental adaptability weight can be expressed by the following formula:

[0094]

[0095] in, represents the environmental adaptability weight; Indicates the adjustment parameters, which are automatically adjusted according to the environmental stability; represents the static weights preset based on expert knowledge, Represents dynamic weights learned based on historical data.

[0096] The multimodal complementary enhancement mechanism addresses the limitations of single-modality perception by dynamically fusing information from different modalities. This mechanism not only improves the accuracy and sensitivity of water quality anomaly detection, but also possesses adaptive learning capabilities, automatically adjusting the importance of each modality based on environmental changes and task requirements.

[0097] Multimodal complementary enhancement can be expressed by the following formula:

[0098]

[0099] in, Indicates the The enhanced eigenvectors of the modes, Indicates the The original eigenvectors of the modes, Indicates the The mode and The enhancement coefficient between modes is used to adjust the mode Modal The enhanced strength can be adaptively learned and the modal weights can be adjusted according to task requirements. Indicates the The mode and The correlation between the modalities is usually calculated using cosine similarity or covariance. Indicates the The original eigenvectors of the modes.

[0100] By employing an attention fusion mechanism with time decay to process features at different time scales and intelligently integrating water quality characteristics at the target time point with those at historical time points, the system accurately captures and predicts changing trends in aquarium water quality. This mechanism uses attention calculations to identify patterns in historical data that are most relevant to the current state. It also automatically adjusts the importance of information at different time points using a decay coefficient, enabling it to distinguish short-term fluctuations from long-term trends. This design not only improves the early detection of water quality anomalies, but also enhances the understanding and prediction accuracy of cyclical changes, effectively avoiding the potential misjudgments that can result from relying solely on current data.

[0101] Temporal attention fusion can be expressed by the following formula:

[0102]

[0103] in, Indicates the target time point Temporal attention fusion features, Indicates a historical point in time. represents the attenuation coefficient, which is usually It decreases with the increase of time, controlling the influence of historical data of different time distances on current decision-making. Generally, the farther the historical data, the smaller its weight. Indicates the water quality characteristics at the target time and the past of the target time The attention mechanism assigns different weights to each historical moment, highlighting the historical information most relevant to the current state.

[0104] Here, the water quality characteristics at the target time point or the water quality characteristics at the historical time point are determined based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector at the corresponding time point.

[0105] According to the present invention, a fish tank water quality monitoring method based on multimodal fusion is provided. The method determines the comprehensive water quality score corresponding to the fish tank water image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature of the target time point, including:

[0106]

[0107] in, Indicates the comprehensive water quality score. represents the normalization function, represents the modal index, Represents the temporal attention fusion feature of the target time point, Indicates the The enhanced eigenvectors of the modes, Represents Different modal indexes, Indicates the The mode and The environmental adaptability weight between the modalities, Indicates the The enhanced eigenvectors of the modes.

[0108] In this embodiment of the present invention, a multimodal fusion method is introduced to quantify water health on a standardized scale. By integrating the independent contributions of three different monitoring modalities (visual analysis of water color, analysis of fish behavioral patterns, and analysis of physical / chemical parameters) and their pairwise interactions, correlations that traditional single-parameter systems may miss are detected. Weighted cross-modal interaction terms enable the system to identify complex patterns, such as synchronized changes in water color and fish behavior, which may indicate serious water quality issues before conventional sensors detect chemical changes. A sigmoid normalization function converts complex multimodal data into an intuitive 0-1 health score.

[0109] The specific comprehensive water quality score can be referred to the above formula, which will not be repeated here.

[0110] In some embodiments, The Sigmoid function is usually used to represent the normalization function, ensuring that the output value range is between 0 and 1, which is easy to understand and compare. The function is expressed as: ,in, For input data.

[0111] In an embodiment of the present invention, i and j Indicates different monitoring modes. In this system:

[0112] i Represents the first mode, with values ​​of 1, 2, and 3; j Represents the second mode, the value is 2 or 3, and is greater than i ;condition j > i Ensure that each pair of modes is calculated only once. 1 represents the water color analysis mode; 2 represents the fish behavior analysis mode; 3 represents the water physical phenomenon analysis mode; specifically, when i is equal to 1, j When is equal to 2, it indicates the interaction between water color mode and fish behavior mode; when i is equal to 1, j When it is equal to 3, it indicates the interaction between the water color mode and the water physical phenomenon mode; when i is equal to 2, j When it is equal to 3, it indicates the interaction between the fish behavior mode and the water physical phenomenon mode.

[0113] This part mainly calculates the individual contribution of each modality. The features of each modality are first enhanced by the temporal attention mechanism, and the sum is calculated to obtain the total independent influence of each modality.

[0114] This part calculates the interaction effects between the modes and captures the synergy between different modes through feature interaction operations. For example, when water color changes and abnormal fish behavior occur at the same time, it may indicate a more serious water quality problem. is the modal interaction weight, which controls the weight parameter of the interaction importance between different modalities and represents the modal i and modal j The importance of interaction between them.

[0115] For example, suppose the system detects the following: the water begins to take on a slight greenish color (early algae growth), fish have a slightly increased breathing rate but normal activity levels, and a small number of persistent bubbles appear at the water surface.

[0116] None of these three items may seem serious individually, but together they may indicate that water quality is beginning to deteriorate. Through the temporal attention mechanism, the system can detect the changing trends of these indicators over the past few days. If the trend persists, the system will issue an early warning even if the current value is still within a safe range. The modal interaction term can capture the correlation between changes in water color and increased fish breathing rate. This combined pattern may be more indicative of water quality problems than any single indicator. Through this comprehensive assessment method, the system can provide a more accurate and predictive water health status score, helping users take preventive measures before water quality problems deteriorate seriously.

[0117] According to a method for monitoring fish tank water quality based on multimodal fusion provided by the present invention, after performing multimodal fusion analysis based on water color feature vectors, fish behavior feature vectors, and water physical phenomenon feature vectors to obtain a comprehensive water quality score corresponding to the fish tank water image, the method further includes:

[0118] Get user query instructions;

[0119] Map user query instructions to a predefined set of intent categories through a pre-trained language model to obtain the user intent category;

[0120] Based on the water color feature vector, fish behavior feature vector, and water physical phenomenon feature vector, vector similarity search is performed in the preset knowledge base to obtain the target historical case;

[0121] According to the cross-modal attention mechanism, the user intention category and the target historical cases are fused to obtain the fused feature vector;

[0122] Based on the water color feature vector, fish behavior feature vector, water physical phenomenon feature vector and target historical cases, vector search is performed in the preset knowledge base to obtain the knowledge base search results;

[0123] Through retrieval enhancement generation, based on the fusion of feature vectors and knowledge base retrieval results, the inference answer results corresponding to the user's query instructions are generated.

[0124] In some embodiments, the present invention also includes a question-and-answer interactive system. The present invention's knowledge base comprises three components: first, fish physiological and ecological knowledge, covering the living conditions, behavioral characteristics, and health standards of different fish species; second, water quality management knowledge, including the safe ranges for various indicators, causes of abnormalities, and treatment methods; and third, historical monitoring data, recording changes in water quality and fish status during system operation. The knowledge base utilizes a graph database for storage, constructing an entity-relationship network and achieving structured representation through ontology models and semantic annotation. Furthermore, a knowledge update mechanism is established to continuously enrich and optimize knowledge content based on system operation, improving the ability to respond to new situations.

[0125] Relying on a pre-trained language model with a deep learning architecture, this system uses a labeled fish tank management conversation dataset for domain adaptation and a multi-label classification framework to map user queries to a predefined set of intent categories, including but not limited to: water quality status queries (such as "How is the water quality?"), fish status queries (such as "Are the goldfish active today?"), historical trend analysis (such as "How has the water quality changed in the past week?"), problem diagnosis (such as "Why aren't the fish eating?"), and maintenance recommendations (such as "Does the water need to be changed?").

[0126] The multimodal feature vectors collected in real time are fused with the knowledge base, primarily using a two-way fusion architecture. First, vector similarity retrieval is used to identify historical cases from the knowledge base that are most similar to the current state. Second, semantic matching is used to identify the intent and focus of the user's question. During the fusion process, a cross-modal attention mechanism is used to dynamically adjust the weights of different information sources to ensure targeted and accurate responses. Furthermore, a causal reasoning module is introduced to infer the possible causes and trends of water quality changes based on current observations and historical knowledge. To improve retrieval efficiency, a hierarchical indexing structure and an approximate nearest neighbor (ANN) algorithm are used to enable rapid retrieval of large-scale knowledge bases.

[0127] Based on the fused feature vectors and knowledge base retrieval results, a Transformer-based decoder architecture is adopted, combined with retrieval-augmented generation (RAG) technology, to provide users with smooth and accurate intelligent question-answering and reasoning.

[0128] Based on the water quality assessment results of multimodal fusion, an early warning threshold is set to determine whether the water quality is abnormal. When the confidence level of a certain abnormal water quality category exceeds the threshold, an early warning is triggered, reminding the user that the water quality is worrying and requires timely attention. In the case of minor anomalies, a prompt message is issued to encourage user attention; in the case of moderate and severe anomalies, a warning message is issued, providing specific treatment suggestions. The early warning trigger threshold is dynamically adjusted according to the fish species, number, and growth stage to avoid false alarms and missed alarms. Users can also query the details of the anomaly through the interactive interface. The data processing equipment also has an indicator light reminder function. When the water quality is good, a green light is displayed. When there are certain abnormalities in the water quality, a yellow light is displayed; when the water quality is seriously abnormal, a red light is displayed to remind the user that the water quality is worrying and requires timely attention.

[0129] The following describes a hardware implementation device of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention.

[0130] refer to Figure 3 , Figure 3 It is a schematic diagram of the hardware implementation device of the multimodal fusion fish tank water quality monitoring method provided by the present invention, which includes a high-definition camera, data processing equipment and cloud.

[0131] This aquarium water quality monitoring method utilizes a high-performance image acquisition device equipped with a high-definition camera, secured to the aquarium's glass front with a suction cup. This device provides basic water quality characterization. The camera captures clear images under standard indoor lighting conditions. Its fixed focal length makes it suitable for monitoring small and medium-sized aquariums. It covers the standard aquarium observation range and maintains stable image quality under normal indoor lighting conditions (minimum 5 Lux). The camera's housing features an IP65 waterproof design, providing splash and dust resistance. Made of environmentally friendly ABS plastic, it is lightweight and flexible. The camera connects to data processing equipment via a USB port for plug-and-play operation, making it easy to install and maintain.

[0132] The data processing device is an embedded device that uses an indicator light to alert the user. Considering its low cost and suitability for home use, it only needs to meet basic image processing and data analysis needs. The data processing device is located below the camera and connected via a USB cable, forming a vertically stacked structure that is easy to install within the limited space around the fish tank.

[0133] This data processing device specializes in visual analysis and is designed for low-power consumption. Under normal monitoring conditions, it performs complete data acquisition and processing every 15 minutes, completing the following tasks: real-time acquisition and storage of high-definition camera image data; execution of computer vision-based water quality anomaly detection algorithms, including water turbidity analysis, surface floating object detection, and water color anomaly identification; local caching of short-term monitoring data and abnormal event images; encrypted upload of processed data and analysis results to the cloud platform via the MQTT protocol; reception and execution of control commands from the cloud platform; support for OTA remote firmware upgrades; and maintenance of local monitoring and early warning functions in the event of network interruptions. The data processing device features an indicator light reminder function that provides alerts based on water quality conditions. This device specializes in image data acquisition and processing, enabling water quality monitoring through multimodal visual analysis technology. It can effectively assess water quality without the need for additional sensors, significantly reducing system complexity and maintenance costs.

[0134] The following describes the fish tank water quality monitoring system based on multimodal fusion provided by the present invention. The fish tank water quality monitoring system based on multimodal fusion described below and the fish tank water quality monitoring method based on multimodal fusion described above can refer to each other.

[0135] refer to Figure 4 , Figure 4 It is a module schematic diagram of the fish tank water quality monitoring system based on multimodal fusion provided by the present invention.

[0136] An acquisition module 401 is configured to acquire a fish tank water image, wherein the fish tank water image includes: a water color image, a fish behavior video, and a water physical phenomenon image;

[0137] The water quality assessment module 402 is used to input the pre-processed fish tank water image into the fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes:

[0138] Perform water color analysis on the water color image to obtain the water color feature vector;

[0139] Perform fish behavior analysis on fish behavior videos to obtain fish behavior feature vectors;

[0140] Performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector;

[0141] Multimodal fusion analysis is performed based on the water color feature vector, fish behavior feature vector, and water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image.

[0142] Specifically, the above-mentioned fish tank water quality monitoring system based on multimodal fusion provided by the present invention can implement all the method steps implemented in the above-mentioned fish tank water quality monitoring method embodiment based on multimodal fusion, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0143] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a fish tank water quality monitoring method based on multimodal fusion, which includes: obtaining a fish tank water body image, wherein the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the preprocessed fish tank water body image into the fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0144] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0145] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fish tank water quality monitoring method based on multimodal fusion provided by the above methods, the method including: obtaining a fish tank water body image, wherein the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the preprocessed fish tank water body image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0146] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fish tank water quality monitoring method based on multimodal fusion provided by the above-mentioned methods, the method comprising: obtaining a fish tank water body image, wherein the fish tank water body image comprises: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the preprocessed fish tank water body image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which comprises: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0148] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A fish tank water quality monitoring method based on multimodal fusion, characterized in that: include: Acquire a fish tank water image, wherein the fish tank water image includes: a water color image, a fish behavior video, and a water physical phenomenon image; The pre-processed fish tank water image is input into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes: Performing water color analysis on the water color image to obtain a water color feature vector; Performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; Performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; Performing a multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image; The performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector includes: The image understanding model is used to extract features of floating objects, abnormal bubbles and suspended particles on the water body physical phenomenon image based on a multi-scale attention mechanism to obtain a feature vector of the water body physical phenomenon.

2. The fish tank water quality monitoring method based on multimodal fusion according to claim 1 is characterized in that: The performing water color analysis on the water color image to obtain a water color feature vector includes: Build a spatiotemporal feature extraction network with a general video understanding model as the backbone network; Pre-training the spatiotemporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatiotemporal feature extraction network; Inserting a trainable parameter adapter module into the pre-trained spatiotemporal feature extraction network for domain fine-tuning to obtain a fine-tuned spatiotemporal feature extraction network; The fine-tuned spatiotemporal feature extraction network is optimized for classification tasks using the Focal Loss function, and a chlorophyll concentration regression task is introduced to obtain a trained spatiotemporal feature extraction network. The water body color image is input into the trained spatiotemporal feature extraction network to obtain a water body color feature vector output by the trained spatiotemporal feature extraction network.

3. The fish tank water quality monitoring method based on multimodal fusion according to claim 1 is characterized in that: The performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector includes: Extracting features from the fish behavior video using a multi-scale object detection network based on an attention mechanism to obtain a multi-level feature map; Extracting features from the multi-level feature map using a deep convolutional neural network to obtain fish body morphological features; Transfer the knowledge of the general video understanding model to the field of fish behavior analysis and fine-tune it to obtain a fish behavior recognition model; The fish behavior recognition model is used to identify the fish body morphological features to obtain a fish behavior feature vector.

4. The fish tank water quality monitoring method based on multimodal fusion according to claim 1 is characterized in that: The multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector is performed to obtain a comprehensive water quality score corresponding to the fish tank water image, including: Determine the environmental adaptability weight based on the static weight preset based on expert knowledge and the dynamic weight learned based on historical data; Performing multimodal complementary enhancement based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain an enhanced feature vector for each modality, wherein the modalities include: water color modality, fish behavior modality, and water physical phenomenon modality; Based on the water quality characteristics of the target time point and the water quality characteristics of the historical time points, the temporal attention fusion feature of the target time point is obtained; Based on the environmental adaptability weight, the enhanced feature vector of each modality and the temporal attention fusion feature of the target time point, a comprehensive water quality score corresponding to the fish tank water image is determined.

5. The fish tank water quality monitoring method based on multimodal fusion according to claim 4 is characterized in that: Determining a comprehensive water quality score corresponding to the fish tank water image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature of the target time point includes: ； in, Indicates the comprehensive water quality score. represents the normalization function, represents the modal index, represents the temporal attention fusion feature of the target time point, Indicates the The enhanced eigenvectors of the modes, Represents Different modal indexes, Indicates the The mode and The environmental adaptability weight between the modalities, Indicates the The enhanced eigenvectors of the modes.

6. The fish tank water quality monitoring method based on multimodal fusion according to claim 1 is characterized in that: After performing multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image, the method further includes: Get user query instructions; Mapping the user query instruction to a predefined set of intent categories through a pre-trained language model to obtain the user intent category; Based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector, a vector similarity search is performed in a preset knowledge base to obtain a target historical case; According to the cross-modal attention mechanism, the user intention category is fused with the target historical case to obtain a fused feature vector; Based on the water body color feature vector, the fish behavior feature vector, the water body physical phenomenon feature vector, and the target historical case, a vector search is performed in a preset knowledge base to obtain a knowledge base search result; Through retrieval enhancement generation, an inference answer result corresponding to the user query instruction is generated based on the fused feature vector and the knowledge base retrieval result.

7. A fish tank water quality monitoring system based on multimodal fusion, characterized in that: include: An acquisition module is used to acquire a fish tank water image, wherein the fish tank water image includes: a water color image, a fish behavior video, and a water physical phenomenon image; A water quality assessment module is used to input the pre-processed fish tank water image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, which includes: Performing water color analysis on the water color image to obtain a water color feature vector; Performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; Performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; Performing a multimodal fusion analysis based on the water color feature vector, the fish behavior feature vector, and the water physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water image; The performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector includes: The image understanding model is used to extract features of floating objects, abnormal bubbles and suspended particles on the water body physical phenomenon image based on a multi-scale attention mechanism to obtain a feature vector of the water body physical phenomenon.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the fish tank water quality monitoring method based on multimodal fusion as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fish tank water quality monitoring method based on multimodal fusion as described in any one of claims 1 to 6 is implemented.

Images