Fish tank water quality monitoring method and system based on multi-modal fusion

Abstract

The invention provides a fish tank water quality monitoring method and system based on multi-modal fusion, which are applied to the field of fish tank water quality monitoring, and the method comprises the following steps: obtaining a fish tank water body image; inputting the preprocessed fish tank water body image into a fish tank water quality evaluation model to obtain a water quality comprehensive score output by the fish tank water quality evaluation model, including: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; and performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector and the water body physical phenomenon feature vector to obtain a water quality comprehensive score corresponding to the fish tank water body image. According to the invention, a more accurate and predictable water quality health state score can be provided.

Description

Technical Field

[0001] The present invention relates to the technical field of fish tank water quality monitoring, and particularly to a fish tank water quality monitoring method and system based on multimodal fusion. Background Art

[0002] At present, the water quality monitoring of home fish tanks mainly relies on manual observation and professional water quality monitoring equipment. Generally, when keeping ornamental fish at home, the water quality condition is judged mainly by observing characteristics such as the color and smell of the water body, or by using monitoring equipment such as pH and ammonia nitrogen.

[0003] However, these traditional methods have many limitations. For example, human eye observation is easily affected by subjective factors, and manual monitoring is prone to missing sudden changes in water quality, making it difficult to give early warnings and handle them in a timely manner, and it is impossible to accurately evaluate water quality indicators; traditional water quality monitoring relies on electrochemical sensors (such as pH meters and dissolved oxygen probes), which have problems such as electrode aging, frequent calibration requirements, and inability to detect microbial contamination. A single modality is difficult to reflect the comprehensive state of water quality.

[0004] It can be seen that the fish tank water quality monitoring methods in the related art have the technical problem of being difficult to reflect the comprehensive state of water quality. Summary of the Invention

[0005] The present invention provides a fish tank water quality monitoring method and system based on multimodal fusion, which is used to solve the defect that the fish tank water quality monitoring method in the prior art is difficult to reflect the comprehensive state of water quality, and to realize providing a more accurate and more predictive water quality health status score.

[0006] The present invention provides a fish tank water quality monitoring method based on multimodal fusion, including the following steps. Obtain an image of the fish tank water body, where the image of the fish tank water body includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; input the preprocessed image of the fish tank water body into a fish tank water quality evaluation model to obtain a water quality comprehensive score output by the fish tank water quality evaluation model, where it includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the water quality comprehensive score corresponding to the image of the fish tank water body.

[0007] A fish tank water quality monitoring method based on multi-modal fusion provided by the present invention. For the water body color image, water body color analysis is performed to obtain a water body color feature vector, including: constructing a spatio-temporal feature extraction network with a general video understanding model as the backbone network; pre-training the spatio-temporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatio-temporal feature extraction network; inserting a trainable parameter adapter module into the pre-trained spatio-temporal feature extraction network for domain fine-tuning to obtain a fine-tuned spatio-temporal feature extraction network; using the Focal Loss function to optimize the classification task of the fine-tuned spatio-temporal feature extraction network and introducing a chlorophyll concentration regression task to obtain a trained spatio-temporal feature extraction network; inputting the water body color image into the trained spatio-temporal feature extraction network to obtain the water body color feature vector output by the trained spatio-temporal feature extraction network.

[0008] A fish tank water quality monitoring method based on multi-modal fusion provided by the present invention. For the fish behavior video, fish behavior analysis is performed to obtain a fish behavior feature vector, including: extracting features from the fish behavior video through a multi-scale object detection network based on an attention mechanism to obtain a multi-level feature map; extracting features from the multi-level feature map through a deep convolutional neural network to obtain fish body shape features; migrating the knowledge of the general video understanding model to the field of fish behavior analysis and performing fine-tuning to obtain a fish behavior recognition model; identifying the fish body shape features through the fish behavior recognition model to obtain a fish behavior feature vector.

[0009] A fish tank water quality monitoring method based on multi-modal fusion provided by the present invention. For the water body physical phenomenon image, water body physical phenomenon analysis is performed to obtain a water body physical phenomenon feature vector, including: extracting features of floating objects on the water surface, abnormal bubbles, and suspended particles from the water body physical phenomenon image through an image understanding model based on a multi-scale attention mechanism to obtain a water body physical phenomenon feature vector.

[0010] A fish tank water quality monitoring method based on multimodal fusion according to the present invention, which performs multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the water quality comprehensive score corresponding to the fish tank water body image, includes: determining the environmental adaptability weight according to the static weight preset based on expert knowledge and the dynamic weight learned based on historical data; performing multimodal complementary enhancement based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the enhanced feature vector of each modality, where the modalities include: water body color modality, fish behavior modality, and water body physical phenomenon modality; performing temporal attention fusion based on the water quality characteristics at the target time point and the water quality characteristics at the historical time point to obtain the temporal attention fusion feature at the target time point; and determining the water quality comprehensive score corresponding to the fish tank water body image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature at the target time point.

[0011] A fish tank water quality monitoring method based on multimodal fusion according to the present invention, which determines the water quality comprehensive score corresponding to the fish tank water body image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature at the target time point, includes: where, represents the water quality comprehensive score, represents the normalization function, represents the modality index, represents the temporal attention fusion feature at the target time point, represents the th enhanced feature vector of the modality, represents different from the modality index, represents the th modality and the th modality of the environmental adaptability weight, represents the th enhanced feature vector of the modality.

[0012] A fish tank water quality monitoring method based on multi-modal fusion provided by the present invention, after performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the water quality comprehensive score corresponding to the fish tank water body image, the method further includes: obtaining a user query instruction; mapping the user query instruction to a predefined set of intention categories through a pre-trained language model to obtain a user intention category; performing vector similarity retrieval in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a target historical case; performing fusion based on the user intention category and the target historical case according to the cross-modal attention mechanism to obtain a fusion feature vector; performing vector retrieval in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector, the water body physical phenomenon feature vector, and the target historical case to obtain a knowledge base retrieval result; generating an inference answer result corresponding to the user query instruction based on the fusion feature vector and the knowledge base retrieval result through retrieval-augmented generation.

[0013] The present invention also provides a fish tank water quality monitoring system based on multi-modal fusion, including the following modules: an acquisition module, configured to acquire a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; a water quality evaluation module, configured to input the preprocessed fish tank water body image into a fish tank water quality evaluation model to obtain a water quality comprehensive score output by the fish tank water quality evaluation model, where it includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the water quality comprehensive score corresponding to the fish tank water body image.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the fish tank water quality monitoring method based on multi-modal fusion as described in any one of the above.

[0015] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the fish tank water quality monitoring method based on multi-modal fusion as described in any one of the above.

[0016] The present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the multimodal fusion-based fish tank water quality monitoring method as described in any one of the above.

[0017] The multimodal fusion-based fish tank water quality monitoring method and system provided by the present invention comprehensively cover multiple key influencing factors of fish tank water quality (such as chemical indicators, biological behaviors, physical states) by simultaneously acquiring water body color images, fish behavior videos, and water body physical phenomenon images; extract feature vectors from water body colors, fish behaviors, and physical phenomena respectively, avoiding the limitations of a single data source and being able to more comprehensively reflect the water quality status; can quickly identify water quality anomalies by analyzing water body color images; can indirectly evaluate the health status of the water body by analyzing fish behavior videos; can further verify the reasons for water quality anomalies by analyzing water body physical phenomenon images (such as bubbles, turbidity, etc.); by combining the above three types of modalities, can cross-verify different data sources to reduce misjudgments and improve the accuracy and reliability of the comprehensive water quality score; through the comprehensive water quality score, users can intuitively understand the current water quality status and timely discover potential problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art one by one. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a flowchart of the multimodal fusion-based fish tank water quality monitoring method provided by the present invention.

[0020] Figure 2 is an overall flowchart of the multimodal fusion-based fish tank water quality monitoring method provided by the present invention.

[0021] Figure 3 is a schematic diagram of the hardware implementation device of the multimodal fusion-based fish tank water quality monitoring method provided by the present invention.

[0022] Figure 4 is a module schematic diagram of the multimodal fusion-based fish tank water quality monitoring system provided by the present invention.

[0023] Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] The keeping of ornamental fish at home has become increasingly popular globally and has become a widely loved leisure hobby. However, due to the lack of professional knowledge and experience, many families have many problems in fish farming management, such as improper water quality monitoring, unreasonable feeding, and untimely disease prevention and control, resulting in damage to the health of ornamental fish and even death. The water quality of the fish tank is a key factor affecting the health of fish. The deterioration of water quality will cause a series of problems such as fish breathing difficulties, decreased immunity, and frequent diseases. Therefore, effective monitoring and management of the water quality of the fish tank are crucial for ensuring the health of ornamental fish.

[0026] Currently, the monitoring of the water quality of home fish tanks mainly relies on manual observation and professional water quality monitoring equipment. Generally, when keeping ornamental fish at home, the water quality condition is judged mainly by observing the characteristics of the water body such as color and smell, or by using monitoring equipment such as pH and ammonia nitrogen. These traditional methods have many limitations: First, human eye observation is easily affected by subjective factors, and manual monitoring is prone to missing sudden changes in water quality, making it difficult to give early warnings and take timely measures, and it is impossible to accurately evaluate water quality indicators; Second, traditional water quality monitoring relies on electrochemical sensors (such as pH meters and dissolved oxygen probes), which have problems such as electrode aging, frequent calibration requirements, and inability to detect microbial contamination. A single modality is difficult to reflect the comprehensive state of water quality; In addition, professional water quality detection equipment is expensive and not suitable for ordinary families. These problems lead to poor effects in keeping ornamental fish at home, difficult to guarantee the health of fish, and high breeding costs.

[0027] To address the above problems, it is necessary to develop a low-cost, easy-to-operate, and intelligent fish tank water quality monitoring system to meet the needs of the majority of home fish-keeping enthusiasts. Based on artificial intelligence technology, the present invention proposes a fish tank water quality monitoring method, system, and device based on multi-modal fusion. By using machine vision and deep learning algorithms, through the combination of analyzing various modalities of fish tank images and videos, it realizes the automatic evaluation and real-time warning of water quality indicators, greatly improving the monitoring efficiency and accuracy. At the same time, by continuously monitoring the changes in fish behavior and appearance, it can detect the signs of fish diseases at an early stage and help users take timely prevention and control measures. The present invention can significantly reduce the risk of fish diseases, extend the service life of the fish tank, save maintenance costs, and thus improve the success rate of keeping ornamental fish at home and the user experience.

[0028] The present invention provides a fish tank water quality monitoring method, system and device based on multimodal fusion, aiming to solve the problems in traditional home ornamental fish water quality monitoring, such as relying on subjective manual observation, incomplete monitoring by single-modal sensors with high costs, difficulty in real-time warning of water quality deterioration and fish health risks, etc.

[0029] The present invention collects fish tank image and video data through a high-definition camera, and conducts in-depth learning analysis on three modalities: water body color, fish behavior and water body physical phenomena respectively. At the same time, three mechanisms, namely environmental adaptability weight adjustment, multimodal complementary enhancement and temporal attention fusion, are adopted for multimodal data fusion. Finally, early warning judgment is carried out based on the water quality health score calculated from the fusion result.

[0030] In addition, a knowledge base and a natural language processing module are equipped to support users to understand the water quality status and treatment suggestions through question-and-answer interaction. Hardware-wise, a suction cup type high-definition camera and a low-power data processor are adopted, which are low-cost and easy to install. The present invention introduces a multimodal fusion interaction mechanism into the field of home fish tank monitoring, replaces traditional sensors through visual analysis, reduces hardware costs, significantly improves monitoring sensitivity and prediction ability, and at the same time integrates lightweight hardware and a knowledge base-driven interaction system to achieve intelligent management with low power consumption and high usability. The present invention can significantly improve the feeding efficiency and success rate of home ornamental fish, and reduce the fish disease risk and maintenance costs.

[0031] Optionally, the multimodal fusion-based fish tank water quality monitoring method of the embodiments of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device. Taking the server to execute the multimodal fusion-based fish tank water quality monitoring method in this embodiment as an example.

[0032] Figure 1 is a schematic flowchart of the multimodal fusion-based fish tank water quality monitoring method provided by the present invention, as Figure 1 shown, the method includes the following steps.

[0033] Step 101, obtain a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image.

[0034] In the embodiments of the present invention, to obtain a high-quality fish tank water body image, the present invention uses a high-definition camera to take real-time pictures of the fish tank. The high-definition camera has excellent imaging performance and can output images and videos with high resolution and high dynamic range.

[0035] According to the monitoring requirements, the high-definition camera supports flexible shooting mode settings, including timed shooting and continuous shooting. Timed shooting can automatically collect images of the fish tank water body at set time intervals, while continuous shooting can continuously record videos during a specific period to capture the dynamic change process of water quality. It can obtain rich enough fish tank water body image samples without disturbing the normal operation of the fish tank, providing a reliable data basis for subsequent water quality analysis.

[0036] Step 102: Input the preprocessed fish tank water body image into the fish tank water quality assessment model to obtain the comprehensive water quality score output by the fish tank water quality assessment model.

[0037] In the embodiment of the present invention, necessary preprocessing is performed on the collected fish tank water body images, such as size normalization, pixel normalization, etc., to meet the format requirements for input to the fish tank water quality assessment model.

[0038] Then, the preprocessed fish tank water body image is input into the water quality assessment model. Through a series of operations such as convolution and pooling, high-level semantic features are extracted. The water quality assessment model finally outputs the confidence scores (comprehensive water quality scores) of the fish tank water body image belonging to each water quality category, reflecting its understanding and judgment of the image content.

[0039] To train the fish tank water quality assessment model, it is necessary to collect a certain-scale fish tank water body image dataset, divide the fish tank water body image dataset into a training set and a test set, perform model training on the training set, and use methods such as cross-validation and grid search to adjust the hyperparameters of the model to obtain the optimal weights and optimize the model performance. The fish tank water body image dataset should cover various common water quality conditions, such as clear, turbid, green water, yellow water, etc., as well as various activity characteristics of fish. Each fish tank water body image sample should be labeled to indicate its true water quality category. To improve the robustness of the model, the fish tank water body image dataset should also include fish tank water body images under changes in factors such as lighting and perspective. To improve the generalization of the model, data augmentation techniques such as random cropping, rotation, and flipping can also be used to expand the sample diversity. Sufficient data volume and reliable annotation quality are the keys to training a high-performance model.

[0040] Among them, step 102 specifically includes the following steps: Step 1021: Perform water body color analysis on the water body color image to obtain a water body color feature vector.

[0041] In some embodiments, a convolutional neural network (CNN) or Vision Transformer (such as Swin Transformer) is used as the backbone network. These models can effectively extract local and global color features of the image.

[0042] Input the preprocessed water body color image into the model, and extract color features through multi-layer convolution or self-attention mechanism. The last layer of the model (usually a fully connected layer or a global pooling layer) outputs a water body color feature vector.

[0043] Step 1022: Perform fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector.

[0044] In some embodiments, preprocess the fish behavior video, and input the preprocessed video frame sequence into the fish behavior analysis model. The model extracts the spatial features (such as fish posture and position) and temporal features (such as swimming trajectory and behavior dynamics) of the video frames layer by layer through spatio-temporal convolution or self-attention mechanism. Finally, a fixed-length feature vector is output, representing the behavior information of the fish.

[0045] Step 1023: Perform water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector.

[0046] Use a convolutional neural network (CNN) or Vision Transformer (such as Swin Transformer) as the backbone network. These models can effectively extract local and global features of the image. Input the preprocessed water body physical phenomenon image into the model, and extract physical phenomenon features through multi-layer convolution or self-attention mechanism. The last layer of the model (usually a fully connected layer or a global pooling layer) outputs a water body physical phenomenon feature vector.

[0047] Step 1024: Perform multi-modal fusion analysis based on the water body color feature vector, fish behavior feature vector, and water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0048] Reference Figure 2 , Figure 2 is the overall process schematic diagram of the fish tank water quality monitoring method based on multi-modal fusion provided by the present invention, including: image acquisition to obtain water body color images, fish behavior videos, and water body physical phenomenon images, image preprocessing, training data set, model training, obtaining optimal weights, water quality analysis, water body color analysis and fine-tuning to obtain a water body color feature vector, fish behavior analysis and fine-tuning to obtain a fish behavior feature vector, water body physical phenomenon analysis and fine-tuning to obtain a water body physical phenomenon feature vector, multi-modal fusion analysis, question-and-answer interaction, single-modal question-and-answer, and early warning reminder.

[0049] In the embodiments of the present invention, the water quality analysis module is divided into three parts, namely the water body color analysis module, the fish behavior analysis module, and the water body physical phenomenon analysis module, to realize a comprehensive intelligent analysis and evaluation of the fish tank water quality state. Through the comprehensive analysis of multi-dimensional data, the accuracy, timeliness, and comprehensiveness of water quality monitoring are effectively improved.

[0050] Through the above steps of the embodiments of the present invention, by simultaneously acquiring the water body color image, fish behavior video, and water body physical phenomenon image, multiple key influencing factors of the fish tank water quality are comprehensively covered (such as chemical indicators, biological behavior, physical state); feature vectors are extracted from the water body color, fish behavior, and physical phenomenon respectively, avoiding the limitations of a single data source and being able to more comprehensively reflect the water quality status; by analyzing the water body color image, water quality anomalies can be quickly identified; by analyzing the fish behavior video, the health status of the water body can be indirectly evaluated; by analyzing the water body physical phenomenon image (such as bubbles, turbidity, etc.), the reasons for water quality anomalies can be further verified; by combining the above three types of modalities, different data sources can be cross-validated to reduce misjudgment and improve the accuracy and reliability of the comprehensive water quality score; through the comprehensive water quality score, users can intuitively understand the current water quality status and timely discover potential problems.

[0051] According to a fish tank water quality monitoring method based on multi-modal fusion provided by the present invention, water body color analysis is performed on the water body color image to obtain a water body color feature vector, including: Construct a spatio-temporal feature extraction network with a general video understanding model as the backbone network; Pre-train the spatio-temporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatio-temporal feature extraction network; Insert a trainable parameter adapter module into the pre-trained spatio-temporal feature extraction network for domain fine-tuning to obtain a fine-tuned spatio-temporal feature extraction network; Use the Focal Loss loss function to optimize the classification task of the fine-tuned spatio-temporal feature extraction network, and introduce a chlorophyll concentration regression task to obtain a trained spatio-temporal feature extraction network; Input the water body color image into the trained spatio-temporal feature extraction network to obtain the water body color feature vector output by the trained spatio-temporal feature extraction network.

[0052] In the embodiments of the present invention, the spatio-temporal feature extraction ability of a general video understanding model (such as Video Swin Transformer, TimeSformer, etc.) is migrated to the field of water body color analysis, and through multi-modal data fusion and domain adaptive fine-tuning, a dedicated model that can accurately identify 6 types of water body colors (clear and transparent, light green, brown / yellow, white turbidity, blue-green, milky white / gray) is constructed.

[0053] For example, using the Video Swin Transformer as the backbone network, a spatio-temporal feature extraction network is constructed to capture the dynamic change characteristics of water bodies. Through a two-stage transfer learning strategy, the basic features are pre-trained on ImageNet-21K and the water area video dataset first, and then a trainable Adapter module is inserted for domain fine-tuning. The Focal Loss is combined to solve the problem of sample imbalance, and at the same time, a chlorophyll concentration regression task is introduced to enhance the model interpretability, resulting in a trained spatio-temporal feature extraction network. The trained spatio-temporal feature extraction network combines visual features and water quality parameters, and outputs color categories and corresponding ecological causes, providing an important basis for subsequent water quality early warning determination.

[0054] The trained spatio-temporal feature extraction network identifies clear and transparent water bodies through its unique light refraction pattern and visibility index. Clear and transparent water bodies usually indicate a healthy and balanced aquatic environment, including appropriate dissolved oxygen levels, minimal suspended solids, and controlled nutrient levels.

[0055] When the trained spatio-temporal feature extraction network detects a light green tint in the water body, it triggers the analysis of the algae growth index. Light green water bodies are usually caused by eutrophication resulting from the combination of excessive nutrients (phosphorus and nitrogen) and sufficient light penetration.

[0056] The trained spatio-temporal feature extraction network identifies brown or yellow water bodies as an indicator of elevated tannin levels or organic matter decomposition. Brown or yellow water bodies are often caused by the decomposition of humus (such as driftwood) or excessive fish food in the water.

[0057] The trained spatio-temporal feature extraction network identifies white turbid water bodies through the reflection and scattering patterns characteristic of high bacterial populations or suspended organic particles. By analyzing the turbidity and pattern, the bacterial concentration level and the severity of the organic load are estimated to alert for timely intervention before reaching the critical threshold.

[0058] When the trained spatio-temporal feature extraction network identifies blue-green water bodies, this usually indicates the presence of a cyanobacteria bloom. Due to the potential toxin production, this situation requires immediate alert to users, and by analyzing the color intensity, distribution pattern, and temporal progression, the development stage of the bloom is evaluated.

[0059] When the trained spatio-temporal feature extraction network identifies milky or gray water bodies, these colors are usually caused by undissolved chemical agents, including fertilizers, pH regulators, or water treatment compounds. Evaluate the color distribution, opacity level, and precipitation characteristics to distinguish between temporary treatment effects and potentially harmful chemical imbalances.

[0060] Through the embodiments of the present invention, a spatio-temporal feature extraction network with a general video understanding model as the backbone network can effectively capture the spatio-temporal features of water body color images, including color distribution, texture changes, etc. By pre-training on a water area video dataset, the model can learn general features related to water bodies; by inserting a trainable parameter adapter module for domain fine-tuning, the model can quickly adapt to the specific scenario of the fish tank water body and improve the accuracy of feature extraction. Using the FocalLoss loss function to optimize the classification task can effectively solve the problem of sample imbalance (such as fewer samples in some color categories) and improve the model's recognition ability for rare categories. By introducing a chlorophyll concentration regression task, the model can not only extract color features but also learn implicit information related to chlorophyll concentration, enhancing the robustness and interpretability of the model.

[0061] According to a fish tank water quality monitoring method based on multimodal fusion provided by the present invention, fish behavior analysis is performed on a fish behavior video to obtain a fish behavior feature vector, including: Feature extraction is performed on the fish behavior video through a multi-scale object detection network based on an attention mechanism to obtain a multi-level feature map; Feature extraction is performed on the multi-level feature map through a deep convolutional neural network to obtain fish body shape features; Transfer the knowledge of the general video understanding model to the field of fish behavior analysis and perform fine-tuning to obtain a fish behavior recognition model; The fish body shape features are recognized through the fish behavior recognition model to obtain a fish behavior feature vector.

[0062] In the embodiments of the present invention, the fish behavior analysis module includes: object detection and individual recognition, feature extraction, and behavior recognition.

[0063] In the object detection link, a multi-scale object detection network based on an attention mechanism is used. This network first extracts a multi-level feature map through a backbone network (such as an improved ResNet or EfficientNet), and then introduces a custom spatial attention and channel attention module. The spatial attention module can dynamically learn the spatial distribution features of fish targets in the image and effectively suppress background noise; the channel attention module can adaptively adjust the importance of different feature channels, enhancing the network's perception ability of the subtle morphological features of fish, so as to achieve precise positioning and recognition of underwater fish individuals. Through the multi-scale feature extraction technology, this model can effectively handle the problems of fish size, posture, and complex background in the underwater environment.

[0064] Deep convolutional neural networks (such as DenseNet or ResNeXt variants) extract fish body morphological features, including body surface texture, fin posture, and gill status. Combining 3D convolution and temporal attention mechanisms to capture the dynamic changes in fish movement patterns. Physiological state feature modeling is carried out through fine-grained region analysis, focusing on health indicators such as respiration, swimming, and body surface conditions. Self-supervised representation learning reduces the dependence on a large amount of labeled data and improves the adaptability of the model in new environments.

[0065] Transfer the knowledge of general video understanding models to the field of fish behavior analysis. Select a general video understanding model for fine-tuning to obtain a large model for fish behavior recognition, and identify the fish body morphological features to obtain the following fish behaviors.

[0066] The fish breathes rapidly: The main features include rapid opening and closing of the gill covers, frequent surfacing, and abnormal swimming patterns. By analyzing the deviation of the respiration rate from the normal benchmark, the fish behavior recognition model determines that this behavior is highly correlated with insufficient dissolved oxygen in the water.

[0067] The fish fins are retracted and the swimming is slow: By accurately tracking the fish movement trajectory and body posture, the subtle changes in fin retraction and slow swimming can be detected. These behavior patterns usually manifest as a significant decrease in swimming speed, slow turning movements, and a decrease in the fin spreading angle. By fusing and analyzing these behavior data with water quality parameters, especially ammonia nitrogen and nitrite levels, the fish behavior recognition model can establish the correlation between behavior changes and excessive toxins, and evaluate the severity of water quality problems according to the degree of behavior abnormality. For example, the fish behavior recognition model determines that the current situation may be due to excessive toxins (such as ammonia or nitrite) in the water.

[0068] The fish body has spots or increased mucus: Specifically trained for changes in the fish body surface condition, it can detect signs such as fish body spots, abnormal increase in mucus secretion, and body color changes. Through high-precision image analysis and temporal comparison, it can distinguish normal body surface changes from abnormal conditions caused by pathogen infections. The fish behavior recognition model can judge the correlation between such behaviors and water quality deterioration. For example, it may be a pathogen infection caused by water quality deterioration.

[0069] Through the embodiments of the present invention, the multi-scale object detection network based on the attention mechanism can simultaneously capture local details (such as fish body postures and actions) and global contexts (such as swimming trajectories and group behaviors) in fish behavior videos. The attention mechanism can dynamically focus on key regions in the video (such as fish body positions and behavior changes), improving the accuracy of feature extraction. Using a deep convolutional neural network to further extract features from multi-level feature maps can effectively capture the morphological features of the fish body (such as fish body contours, postures, and actions). Transferring the knowledge of general video understanding models (such as Video Swin Transformer or I3D) to the field of fish behavior analysis and leveraging their powerful spatio-temporal feature extraction capabilities can quickly adapt to new tasks. By fine-tuning on fish behavior datasets, the model can learn specific features related to fish behaviors, improving the accuracy and robustness of behavior recognition.

[0070] According to a method for monitoring the water quality of an aquarium based on multi-modal fusion provided by the present invention, water physical phenomenon analysis is performed on water physical phenomenon images to obtain water physical phenomenon feature vectors, including: The water physical phenomenon feature vectors are obtained by using an image understanding model to extract features of floating objects, abnormal bubbles, and suspended particles in the water physical phenomenon images based on a multi-scale attention mechanism.

[0071] In the embodiments of the present invention, based on an image understanding model (such as CLIP, DINOv2), end-to-end intelligent diagnosis of floating objects, abnormal bubbles, and suspended substances is achieved through fine-grained visual feature decoupling and physical property reasoning. For example, with ViT-Hybrid (combining CNN and Transformer) as the core, morphological and dynamic features of floating objects on the water surface (fish food residues, excrement clumps), abnormal bubbles (diameter > 2 mm and staying > 10 seconds), and suspended particles (particle size > 50 μm) are extracted through a multi-scale attention mechanism.

[0072] Among them, the floating objects or sediments are fish food residues or excrement, which may cause water quality deterioration. Bubble stay: For abnormal bubbles with a diameter > 2 mm and staying > 10 seconds, it may be judged that the surface tension of the water body is abnormal, usually related to excessive organic matter. Whether there are suspended substances in the water: When the particle size in the water quality > 50 μm, it is judged that the turbidity is high, and the filtration system may need to be cleaned.

[0073] Through the embodiments of the present invention, local details and global distributions of water physical phenomena are captured through the attention mechanism and multi-scale feature extraction. Floating objects, abnormal bubbles, and suspended particles on the water surface are accurately detected, and their key features are extracted. The generated water physical phenomenon feature vectors can comprehensively represent the physical state of the water body, providing important inputs for subsequent water quality assessment.

[0074] A fish tank water quality monitoring method based on multi-modal fusion provided by the present invention performs multi-modal fusion analysis based on a water body color feature vector, a fish behavior feature vector, and a water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image, including: Determine the environmental adaptability weight according to the static weight preset based on expert knowledge and the dynamic weight learned from historical data; Perform multi-modal complementary enhancement based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain an enhanced feature vector for each modality, where the modalities include: the water body color modality, the fish behavior modality, and the water body physical phenomenon modality; Perform temporal attention fusion based on the water quality characteristics at the target time point and the water quality characteristics at the historical time point to obtain the temporal attention fusion feature at the target time point; Determine the comprehensive water quality score corresponding to the fish tank water body image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature at the target time point.

[0075] In the embodiment of the present invention, by introducing an environmental adaptability weight adjustment mechanism, the multi-modal fish tank water quality monitoring system can intelligently balance prior knowledge and historical data, rely more on expert experience when the environment is stable, and pay more attention to the data learning results when the environment changes, thereby overcoming the limitations of the traditional fixed weight system, improving the adaptability, generalization ability, and reliability of the system, realizing adaptive decision-making, reducing manual intervention, and providing more intelligent technical support for fish tank water quality monitoring.

[0076] The environmental adaptability weight can be expressed by the following formula: Wherein, represents the environmental adaptability weight; represents an adjustment parameter, which is automatically adjusted according to environmental stability; represents the static weight preset based on expert knowledge, represents the dynamic weight learned from historical data.

[0077] The multi-modal complementary enhancement mechanism solves the problem of the limitation of single-modal perception by dynamically fusing information from different modalities. This mechanism can not only improve the accuracy and sensitivity of water quality anomaly detection, but also has the ability of adaptive learning, and can automatically adjust the importance weights of each modality according to environmental changes and task requirements.

[0078] The multi-modal complementary enhancement can be expressed by the following formula: Wherein, represents the The enhanced feature vector of a modality, represents the original feature vector of the th modality, represents the enhancement coefficient between the th modality and the th modality, which is used to adjust the enhancement intensity of the pair of modalities and can be adaptively learned to adjust the modality weights according to the task requirements. represents the correlation between the th modality and the th modality, which is usually calculated using cosine similarity or covariance. represents the original feature vector of the th modality.

[0079] By adopting an attention fusion mechanism with time decay, features at different time scales are processed. By intelligently integrating the water quality features at the target time point and the water quality features at historical time points, the accurate capture and prediction of the changing trend of the fish tank water quality are achieved. This mechanism uses attention calculation to identify the patterns most relevant to the current state in historical data, and at the same time automatically adjusts the importance of information at different time points through the decay coefficient, being able to distinguish short-term fluctuations and long-term trends. This design not only improves the early detection ability of water quality anomalies but also enhances the understanding and prediction accuracy of periodic changes, effectively avoiding misjudgments that may be caused by relying solely on current data.

[0080] The temporal attention fusion can be expressed by the following formula: where, represents the temporal attention fusion feature at the target time point , represents the historical time point, represents the decay coefficient, which usually decreases with the increase of , controlling the influence degree of historical data at different time distances on the current decision-making. Generally, the weights of historical data farther away are smaller. represents the attention relationship between the water quality feature at the target time point and the water quality features at the historical time points moments in the past of the target time point. The attention mechanism assigns different weights to each historical moment, highlighting the historical information most relevant to the current state.

[0081] Here, the water quality feature at the target time point or the water quality feature at the historical time point is jointly determined according to the water body color feature vector, fish behavior feature vector, and water body physical phenomenon feature vector corresponding to the respective time points.

[0082] A fish tank water quality monitoring method based on multi-modal fusion according to the present invention determines a comprehensive water quality score corresponding to a fish tank water body image based on an environmental adaptability weight, an enhanced feature vector of each modality, and a temporal attention fusion feature at a target time point, including: Wherein, represents the comprehensive water quality score, represents a normalization function, represents a modality index, represents the temporal attention fusion feature at the target time point, represents the th enhanced feature vector of a modality, represents different from modality index, represents the th modality and the th modality of the environmental adaptability weight between, represents the th enhanced feature vector of a modality.

[0083] In an embodiment of the present invention, a multi-modal fusion method is introduced to quantify water quality health on a standardized scale. By integrating the independent contributions of three different monitoring modalities (visual analysis of water body color, analysis of fish behavior patterns, and physical / chemical parameter analysis) and their pairwise interactions, correlations that may be overlooked by traditional single-parameter systems can be detected. The weighted cross-modal interaction terms enable the system to identify complex patterns, such as synchronous changes in water body color and fish behavior, which may indicate serious water quality problems before chemical changes are detected by conventional sensors. The Sigmoid normalization function converts complex multi-modal data into an intuitive 0-1 health score.

[0084] The specific comprehensive water quality score can refer to the above formula and will not be elaborated here.

[0085] In some embodiments, The normalization function usually uses the Sigmoid function to ensure that the output value range is between 0 and 1, which is convenient for understanding and comparison. The function is expressed as: , wherein, is the input data.

[0086] In an embodiment of the present invention, i and j represent different monitoring modalities. In this system: i represents the first modality, taking values of 1, 2, 3; j represents the second modality, taking values of 2 or 3 and greater than i ; conditionj > i Ensure that each pair of modalities is only calculated once. Here, 1 represents the water body color analysis modality; 2 represents the fish behavior analysis modality; 3 represents the water body physical phenomenon analysis modality. Specifically, when i equals 1, j equals 2, it represents the interaction between the water body color modality and the fish behavior modality; when i equals 1, j equals 3, it represents the interaction between the water body color modality and the water body physical phenomenon modality; when i equals 2, j equals 3, it represents the interaction between the fish behavior modality and the water body physical phenomenon modality.

[0087] This part mainly calculates the individual contributions of each modality. The features of each modality are first enhanced by the temporal attention mechanism, and the sum is obtained to get the total independent influence of each modality This part calculates the interaction effects between modalities, and captures the synergistic effects between different modalities through feature interaction operations. For example, when the water body color change and the abnormal fish behavior appear simultaneously, it may indicate a more serious water quality problem. Among them is the modality interaction weight, which is a weight parameter controlling the importance of the interaction between different modalities, representing the modality i and the modality j the importance of the interaction between them.

[0088] For example, assume the system detects the following situation: the water body color begins to show a slight green (early growth of algae), the fish breathing frequency increases slightly, but the activity level is normal, and there are a small number of persistent bubbles on the water surface.

[0089] Individually, these three items may not be serious, but combined, they may indicate that the water quality is starting to deteriorate. Through the temporal attention mechanism, the system can discover the change trends of these indicators in the past few days. If the trend continues, even if the current values are still within the safe range, the system will give an early warning. The modality interaction term can capture the correlation between the water body color change and the increase in fish breathing frequency, and this combined pattern may be more indicative of water quality problems than individual indicators. Through this comprehensive evaluation method, the system can provide a more accurate and predictive water quality health status score to help users take preventive measures before the water quality problem deteriorates seriously.

[0090] According to a fish tank water quality monitoring method based on multi-modal fusion provided by the present invention, after performing multi-modal fusion analysis based on the water body color feature vector, fish behavior feature vector, and water body physical phenomenon feature vector to obtain the water quality comprehensive score corresponding to the fish tank water body image, the method further includes: Obtain a user query instruction; Map user query instructions to a predefined set of intent categories through a pre-trained language model to obtain the user intent category; Based on the water color feature vector, fish behavior feature vector, and water physical phenomenon feature vector, vector similarity search is performed in the preset knowledge base to obtain the target historical case; According to the cross-modal attention mechanism, the user intention category and the target historical cases are fused to obtain the fused feature vector; Based on the water color feature vector, fish behavior feature vector, water physical phenomenon feature vector and target historical cases, vector search is performed in the preset knowledge base to obtain the knowledge base search results; Through retrieval enhancement generation, based on the fusion of feature vectors and knowledge base retrieval results, the inference answer results corresponding to the user's query instructions are generated.

[0091] In some embodiments, the present invention also includes a question-and-answer interactive system. The present invention's knowledge base comprises three components: first, fish physiological and ecological knowledge, covering the living conditions, behavioral characteristics, and health standards of different fish species; second, water quality management knowledge, including the safe ranges for various indicators, causes of abnormalities, and treatment methods; and third, historical monitoring data, recording changes in water quality and fish status during system operation. The knowledge base utilizes a graph database for storage, constructing an entity-relationship network and achieving structured representation through ontology models and semantic annotation. Furthermore, a knowledge update mechanism is established to continuously enrich and optimize knowledge content based on system operation, improving the ability to respond to new situations.

[0092] Relying on a pre-trained language model with a deep learning architecture, this system uses a labeled fish tank management conversation dataset for domain adaptation and a multi-label classification framework to map user queries to a predefined set of intent categories, including but not limited to: water quality status queries (such as "How is the water quality?"), fish status queries (such as "Are the goldfish active today?"), historical trend analysis (such as "How has the water quality changed in the past week?"), problem diagnosis (such as "Why aren't the fish eating?"), and maintenance suggestions (such as "Does the water need to be changed?").

[0093] The multimodal feature vectors collected in real time are fused with the knowledge base, primarily using a two-way fusion architecture. First, vector similarity retrieval is used to identify historical cases from the knowledge base that are most similar to the current state. Second, semantic matching is used to identify the intent and focus of the user's question. During the fusion process, a cross-modal attention mechanism is used to dynamically adjust the weights of different information sources to ensure targeted and accurate responses. Furthermore, a causal reasoning module is introduced to infer the possible causes and trends of water quality changes based on current observations and historical knowledge. To improve retrieval efficiency, a hierarchical indexing structure and an approximate nearest neighbor (ANN) algorithm are used to enable rapid retrieval of large-scale knowledge bases.

[0094] Based on the fused feature vectors and the knowledge base retrieval results, a Transformer-based decoder architecture is adopted, combined with the Retrieval-Augmented Generation (RAG) technology, to provide users with smooth and information-accurate intelligent question answering and reasoning.

[0095] Based on the water quality assessment results of multimodal fusion, warning thresholds are set to judge whether the water quality is abnormal. When the confidence level of a certain abnormal water quality category exceeds the threshold, a warning prompt is triggered to remind the user that the water quality is a cause for concern and needs to be dealt with in a timely manner. A prompt message is sent for minor abnormalities, advising the user to pay attention; a warning message is sent for moderate and severe abnormalities, providing specific treatment suggestions; the warning trigger threshold is dynamically adjusted according to the fish species, quantity, and growth stage to avoid false alarms and missed alarms. Users can also query the details of the abnormality through the interactive interface. At the same time, the data processing device has an indicator light reminder function. When the water quality is good, a green light is displayed. When there is a certain abnormality in the water quality, a yellow light is displayed; when there is a severe abnormality in the water quality, a red light is displayed to remind the user that the water quality is a cause for concern and needs to be dealt with in a timely manner.

[0096] The following describes the hardware implementation device of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention.

[0097] Reference Figure 3 , Figure 3 is a schematic diagram of the hardware implementation device of the fish tank water quality monitoring method based on multimodal fusion provided by the present invention, which includes a high-definition camera, a data processing device, and the cloud.

[0098] The fish tank water quality monitoring method uses a high-performance image acquisition device equipped with a high-definition camera, which is fixed on the front glass of the fish tank by a suction cup method to meet the basic water quality feature recognition requirements. The camera can obtain clear images under standard indoor lighting conditions. The camera has a fixed focal length design, which is suitable for monitoring small and medium-sized fish tanks; it covers the standard fish tank observation range and can maintain stable imaging quality under normal indoor lighting conditions (not less than 5 Lux). The camera housing adopts an IP65 waterproof design, which can prevent splashing water and dust. The body is made of environmentally friendly ABS plastic material, with a light weight and flexible installation. The camera is connected to the data processing device through a USB interface, which is plug-and-play and easy to install and maintain.

[0099] The data processing device is an embedded device and reminds the user through an indicator light. Considering the requirements of low cost and suitability for home use, it only needs to meet the basic image processing and data analysis needs. The data processing device is located below the camera and is connected by a USB cable to form a vertically stacked structure, which is convenient for installation in the limited space around the fish tank.

[0100] This data processing device focuses on visual analysis functions and is designed in a low-power mode. In the normal monitoring state, it conducts a complete data collection and processing every 15 minutes and can perform the following tasks: real-time collect and store high-definition camera image data; execute water quality anomaly detection algorithms based on computer vision, including water turbidity analysis, detection of floating objects on the water surface, and identification of abnormal water color; locally cache short-term monitoring data and images of abnormal events; encrypt and upload the processed data and analysis results to the cloud platform through the MQTT protocol; receive and execute control instructions from the cloud platform; support OTA remote firmware upgrade; maintain local monitoring and warning functions in case of network interruption. The data processing device has an indicator light reminder function to give reminders according to the water quality situation. This device focuses on the collection and processing of image data, realizes water quality monitoring through multi-modal visual analysis technology, and can effectively evaluate the water quality condition without additional sensors, significantly reducing the system complexity and maintenance cost.

[0101] The following describes the fish tank water quality monitoring system based on multi-modal fusion provided by the present invention. The fish tank water quality monitoring system based on multi-modal fusion described below can be mutually corresponded and referred to the fish tank water quality monitoring method based on multi-modal fusion described above.

[0102] Reference Figure 4 , Figure 4 is a schematic diagram of the modules of the fish tank water quality monitoring system based on multi-modal fusion provided by the present invention.

[0103] An acquisition module 401 is used to acquire fish tank water body images, where the fish tank water body images include: water body color images, fish behavior videos, and water body physical phenomenon images; A water quality evaluation module 402 is used to input the preprocessed fish tank water body images into a fish tank water quality evaluation model to obtain a comprehensive water quality score output by the fish tank water quality evaluation model, where it includes: Conduct water body color analysis on the water body color images to obtain water body color feature vectors; Conduct fish behavior analysis on the fish behavior videos to obtain fish behavior feature vectors; Conduct water body physical phenomenon analysis on the water body physical phenomenon images to obtain water body physical phenomenon feature vectors; Conduct multi-modal fusion analysis based on the water body color feature vectors, fish behavior feature vectors, and water body physical phenomenon feature vectors to obtain a comprehensive water quality score corresponding to the fish tank water body images.

[0104] Specifically, the above-mentioned fish tank water quality monitoring system based on multi-modal fusion provided by the present invention can implement all the method steps implemented by the above-mentioned embodiments of the fish tank water quality monitoring method based on multi-modal fusion and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0105] Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention. As Figure 5 shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute a fish tank water quality monitoring method based on multimodal fusion. The method includes: obtaining a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the preprocessed fish tank water body image into a fish tank water quality evaluation model to obtain a comprehensive water quality score output by the fish tank water quality evaluation model, where it includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multimodal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0106] In addition, when the logical instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-modal fusion-based fish tank water quality monitoring method provided by the above-mentioned various methods. The method includes: obtaining a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the pre-processed fish tank water body image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, where it includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0108] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the multi-modal fusion-based fish tank water quality monitoring method provided by the above-mentioned various methods. The method includes: obtaining a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; inputting the pre-processed fish tank water body image into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, where it includes: performing water body color analysis on the water body color image to obtain a water body color feature vector; performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the fish tank water body image.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0110] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fish tank water quality monitoring method based on multimodal fusion, characterized in that, Including: Obtain an image of the fish tank water body, where the image of the fish tank water body includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; Input the preprocessed image of the fish tank water body into a fish tank water quality assessment model to obtain a comprehensive water quality score output by the fish tank water quality assessment model, where it includes: Perform water body color analysis on the water body color image to obtain a water body color feature vector; Perform fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; Perform water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; Perform multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the image of the fish tank water body.

2. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 1, wherein The performing water body color analysis on the water body color image to obtain a water body color feature vector includes: Construct a spatio-temporal feature extraction network with a general video understanding model as the backbone network; Pre-train the spatio-temporal feature extraction network based on a preset water area video dataset to obtain a pre-trained spatio-temporal feature extraction network; Insert a trainable parameter adapter module into the pre-trained spatio-temporal feature extraction network for domain fine-tuning to obtain a fine-tuned spatio-temporal feature extraction network; Use the Focal Loss function to optimize the classification task of the fine-tuned spatio-temporal feature extraction network and introduce a chlorophyll concentration regression task to obtain a trained spatio-temporal feature extraction network; Input the water body color image into the trained spatio-temporal feature extraction network to obtain a water body color feature vector output by the trained spatio-temporal feature extraction network.

3. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 1, characterized in that, The performing fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector includes: Extract features from the fish behavior video through a multi-scale object detection network based on the attention mechanism to obtain a multi-level feature map; Extract features from the multi-level feature map through a deep convolutional neural network to obtain fish body morphology features; Transfer the knowledge of the general video understanding model to the field of fish behavior analysis and perform fine-tuning to obtain a fish behavior recognition model; Identify the fish body morphology features through the fish behavior recognition model to obtain a fish behavior feature vector.

4. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 1, characterized in that The performing water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector includes: Extract features of floating objects on the water surface, abnormal bubbles, and suspended particles from the water body physical phenomenon image through an image understanding model based on a multi-scale attention mechanism to obtain a water body physical phenomenon feature vector.

5. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 1, wherein The performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a comprehensive water quality score corresponding to the image of the fish tank water body includes: Determine an environmental adaptability weight according to a static weight preset based on expert knowledge and a dynamic weight learned from historical data; Perform multi-modal complementary enhancement based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain an enhanced feature vector for each modality, where the modalities include: the water body color modality, the fish behavior modality, and the water body physical phenomenon modality; Perform temporal attention fusion based on the water quality characteristics at the target time point and the water quality characteristics at the historical time point to obtain the temporal attention fusion feature at the target time point; Determine the comprehensive water quality score corresponding to the fish tank water body image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature at the target time point.

6. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 5, characterized in that, The determining the comprehensive water quality score corresponding to the fish tank water body image based on the environmental adaptability weight, the enhanced feature vector of each modality, and the temporal attention fusion feature at the target time point includes: Among them, represents the comprehensive water quality score, represents the normalization function, represents the modal index, represents the temporal attention fusion feature at the target time point, represents the enhanced feature vector of the th modality, represents a modal index different from represents the environmental adaptability weight between the th modality and the th modality, represents the enhanced feature vector of the th modality.

7. The method for monitoring the water quality of an aquarium based on multimodal fusion according to claim 1, wherein After performing multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the comprehensive water quality score corresponding to the fish tank water body image, the method further includes: Obtain a user query instruction; Map the user query instruction to a predefined set of intent categories through a pre-trained language model to obtain the user intent category; Perform vector similarity retrieval in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain a target historical case; Based on the cross-modal attention mechanism, fuse the user intent category and the target historical case to obtain a fused feature vector; Perform vector retrieval in a preset knowledge base based on the water body color feature vector, the fish behavior feature vector, the water body physical phenomenon feature vector, and the target historical case to obtain a knowledge base retrieval result; Generate an inference answer result corresponding to the user query instruction based on the fused feature vector and the knowledge base retrieval result through retrieval-augmented generation.

8. A fish tank water quality monitoring system based on multimodal fusion, characterized in that, including: An acquisition module for acquiring a fish tank water body image, where the fish tank water body image includes: a water body color image, a fish behavior video, and a water body physical phenomenon image; A water quality assessment module for inputting the preprocessed fish tank water body image into a fish tank water quality assessment model to obtain the comprehensive water quality score output by the fish tank water quality assessment model, where it includes: Perform water body color analysis on the water body color image to obtain a water body color feature vector; Perform fish behavior analysis on the fish behavior video to obtain a fish behavior feature vector; Perform water body physical phenomenon analysis on the water body physical phenomenon image to obtain a water body physical phenomenon feature vector; Perform multi-modal fusion analysis based on the water body color feature vector, the fish behavior feature vector, and the water body physical phenomenon feature vector to obtain the comprehensive water quality score corresponding to the fish tank water body image.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the fish tank water quality monitoring method based on multi-modal fusion according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multimodal fusion-based fish tank water quality monitoring method according to any one of claims 1 to 7.

Images