Fish and vegetable symbiotic water quality prediction and regulation system based on multi-modal data fusion

Abstract

The invention relates to the technical field of intelligent agriculture, and discloses a fish and vegetable symbiotic water quality prediction and regulation system based on multi-modal data fusion. The system comprises a multi-modal data sensing module, a feature extraction module, a multi-modal feature fusion module, a water quality prediction model construction module, an intelligent decision module, an equipment control module, an execution effect evaluation module, an online learning and optimization module and a visual interaction module. According to the invention, the limitation of a single sensor is broken through, and multi-dimensional and cross-modal system state sensing is realized; based on the context perception feature vector, a mixed time sequence model is adopted for multi-step prediction, and active early warning of water quality changes is achieved; an expert rule base is combined with an optimization algorithm to generate a globally collaborative intelligent regulation and control strategy; a complete intelligent closed loop of decision-execution-evaluation-learning is established by evaluating the actual effect of each regulation and control action and driving strategy self-optimization and online fine adjustment of a prediction model.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture technology, and more specifically to a system for predicting and regulating water quality in aquaponics based on multimodal data fusion. Background Technology

[0002] Aquaponics, as a sustainable agricultural production model that combines aquaculture and hydroponics, can achieve the environmental goal of raising fish without changing the water and growing vegetables without applying fertilizer through the material cycle within the ecosystem.

[0003] However, the stable and efficient operation of this system highly depends on the precise management and control of water quality. Ammonia nitrogen, nitrite, dissolved oxygen, and pH are key water quality parameters, which are interconnected and dynamically change, directly affecting fish health and plant growth. Currently, water quality management in aquaponics systems mainly relies on two methods: one is automated control based on traditional sensors and thresholds. These systems monitor water quality by deploying physicochemical sensors such as pH, dissolved oxygen, and temperature, and control it using preset fixed thresholds; the other is the introduction of data-driven predictive models. To overcome lag, some studies have begun to attempt to use time-series predictive models to predict water quality parameters.

[0004] The existing technology also has the following drawbacks: Limited information dimensions and inability to detect biological precursors: Existing technologies rely only on a limited number of physical and chemical sensors, which cannot obtain key biological information such as fish behavior and plant status, resulting in delayed early warnings and low system reliability.

[0005] Passive response and severe lag: Existing systems are only in a monitoring-response mode, and only trigger actions when the water quality has deteriorated to a critical point. They cannot predict future trends and are always in a passive remedial state.

[0006] The control measures are isolated and lack coordination: the execution units of existing systems, such as oxygenation and feeding, are usually controlled independently, lacking a global strategy for coordinated optimization, which leads to the mutual cancellation of control effects or waste of resources.

[0007] The system is static and rigid, lacking self-learning ability: existing technologies rely on preset fixed rules or models, and cannot be optimized based on actual results. After system deployment, performance cannot be improved, and the system cannot adapt to environmental changes.

[0008] Therefore, multi-dimensional perception, proactive early warning, global collaborative control, and a complete intelligent closed loop are needed to solve the above problems. Summary of the Invention

[0009] In order to overcome the above-mentioned defects of the prior art, the present invention provides a water quality prediction and control system for aquaponics based on multimodal data fusion, so as to solve the problems existing in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a water quality prediction and control system for aquaponics based on multimodal data fusion, comprising: Multimodal data sensing module: used to deploy sensor arrays to collect multimodal data of the aquaponics system in real time, including physicochemical modal data, optical modal data and acoustic modal data; Feature extraction module: Used to process the collected multimodal data and extract features from the processed data to obtain multimodal feature sequences, including water quality time series data features, fish behavior features, plant physiological features and acoustic data features; Multimodal feature fusion module: Used to deeply fuse the multimodal feature sequences using an attention-based model to construct a context-aware feature vector that can comprehensively describe the health status of the aquaponics system; Water quality prediction model construction module: used to construct a hybrid water quality prediction model using a hybrid time-series deep learning model, perform multi-step prediction of key water quality parameters through the context-aware feature vector, and output the prediction results; Intelligent decision-making module: This module incorporates an expert rule base and optimization algorithms to generate the optimal collaborative control strategy based on the water quality prediction results and the current system status. Equipment control module: Used to receive the optimal coordinated control strategy, convert it into control instructions for specific execution devices, and drive the execution devices to change the system state after verification; Implementation effect evaluation module: Used to calculate the control effect coefficient and generate an evaluation report by analyzing the deviation between the actual change and the expected change of the target variable during the set observation period after the execution of the equipment; Online learning and optimization module: It is used to receive feedback data from the performance evaluation module, adaptively optimize decision-making strategies through reinforcement learning algorithms, trigger online fine-tuning of the water quality prediction model, and dynamically evolve the expert rule base. Visual interaction module: used to provide a global system status dashboard, historical data query, remote manual control of equipment, and early warning information push functions.

[0011] The technical effects and advantages of this invention are as follows: 1. This invention breaks through the limitations of traditional systems that rely solely on physicochemical sensors. By integrating visual characteristics of fish behavior, physiological and morphological characteristics of plants, and acoustic characteristics, it constructs a more comprehensive health profile of the system, which can capture biological behavioral precursors before water quality deterioration, providing earlier and more reliable evidence for early warning.

[0012] 2. Based on deeply fused context-aware feature vectors, this invention uses an advanced hybrid temporal deep learning model for multi-step prediction, which can accurately predict the future trends of key water quality parameters, thereby significantly advancing the control window, transforming passive remediation into active intervention, and effectively avoiding the risk of system collapse.

[0013] 3. This invention, through a decision-making mechanism that combines an embedded expert rule base with an optimization algorithm, can comprehensively consider multiple control objectives and output the optimal strategy for coordinated action of multiple execution devices, thus solving the problem of strategy conflict or resource waste that may be caused by single-point control in traditional methods.

[0014] 4. This invention quantifies the actual effect of each regulatory action and utilizes reinforcement learning to drive self-optimization and online fine-tuning of the prediction model, enabling the system to continuously learn from practical experience and evolve its performance over time. It possesses adaptive and self-learning capabilities that traditional static systems lack. Attached Figure Description

[0015] Figure 1 This is a structural block diagram of the present invention.

[0016] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The aquaponics water quality prediction and control system based on multimodal data fusion involved in the present invention is not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Reference Figure 1 This invention provides a water quality prediction and control system for aquaponics based on multimodal data fusion, including a multimodal data perception module, a feature extraction module, a multimodal feature fusion module, a water quality prediction model construction module, an intelligent decision-making module, an equipment control module, an execution effect evaluation module, an online learning and optimization module, and a visualization interaction module.

[0019] Reference Figure 2 The specific implementation steps of the present invention include the following steps: S1. Deploy a sensor array through a multimodal data sensing module to collect multimodal data of the aquaponics system in real time, including physicochemical modal data, optical modal data and acoustic modal data.

[0020] It should be specifically noted that the multimodal data sensing module specifically includes: Water quality sensor array: Deploy immersion or flow-through sensors to monitor water bodies in real time and collect physicochemical modal data, including pH value, dissolved oxygen concentration, temperature, conductivity and redox potential of the water body; Optical acquisition unit: Deployed underwater in the fishpond, including waterproof cameras, fixed-point cameras targeting plant roots, and wide-angle cameras monitoring the fishpond surface, to capture high-definition video streams and still images, and collect optical modal data, specifically including: Underwater images: used to analyze fish behavior, such as activity level, feeding desire, and whether they surface for air, with surfacing being a direct indication of oxygen deficiency; Water surface images: Analyze fish feeding behavior to determine if there is any uneaten food; Plant images: By analyzing the color and shape of plant leaves through image recognition, the health status of plants can be determined, indirectly reflecting water quality.

[0021] Acoustic acquisition unit: A high-frequency underwater microphone deployed underwater to collect acoustic modal data including fish activity, feeding, and equipment operation.

[0022] S2. The collected multimodal data is processed by the feature extraction module, and the processed data is used to extract features to obtain multimodal feature sequences, including water quality time series data features, fish behavior features, plant physiological features and acoustic data features.

[0023] It should be specifically noted that the processing of the collected multimodal data includes data cleaning, data alignment, and normalization. The data cleaning uses median filtering and sliding window detection to identify and remove outliers caused by sensor interruptions or interference, and uses linear interpolation or forward imputation to repair missing data. The data alignment and normalization uses a unified timestamp as a benchmark to perform time-series alignment of multi-source data with different acquisition frequencies, and uses the Z-Score normalization method to scale the data to the same dimension to eliminate the influence of dimensions.

[0024] The specific features of the water quality time-series data include: Statistical characteristics: Calculate the mean, variance, and trend slope within the sliding window; Periodic characteristics: Extract diurnal cycle patterns and identify abnormal deviations; for example, plants increase pH during the day through photosynthesis and decrease pH at night through respiration. Event triggering characteristics: Treating feeding as an important event marker, calculating the rate of change of water quality parameters before and after feeding, and quantifying the impact of feeding behavior.

[0025] The specific behavioral characteristics of the fish include: Based on the object detection model, fish are identified from the video stream, and the following calculations are performed: Average swimming speed: A decrease in the average speed of the group is an early sign of hypoxia or ammonia poisoning; Group aggregation: Calculate the center point of the fish group and analyze the variance of the distance between the individual fish and the center. Abnormal aggregation indicates stress. Surface activity frequency: The number of times a fish's mouth touches the water surface per unit of time. Fish surfacing is a direct quantitative indicator of oxygen deficiency.

[0026] The plant physiological characteristics specifically include: Color space analysis: Plant leaf color histogram based on HSV color space is used to detect nutrient deficiency symptoms such as yellowing and whitening; Texture features: Leaf texture based on local binary pattern or gray-level co-occurrence matrix, used to determine leaf dehydration or disease; Morphological characteristics: Morphological characteristics based on leaf spread and profile curvature are used to quantify whether the leaves are wilting.

[0027] The acoustic data features specifically include: Mel frequency cepstral coefficients: used to extract the activity level of fish; Spectral centroid and spectral roll-off point: describe the brightness and energy distribution of a sound, used to distinguish between normal swimming sounds and the rapid waggle sound when startled; Sound pressure level temporal variation: Analyze the variation pattern of sound pressure level in a specific frequency band to identify abnormally high-pitched sounds.

[0028] S3. Through the multimodal feature fusion module, an attention-based model is used to deeply fuse the multimodal feature sequences to construct a context-aware feature vector that can comprehensively describe the health status of the aquaponics system.

[0029] It should be specifically noted that the construction steps of the context-aware feature vector are as follows: A1. Input embedding and position encoding; The input feature sequences from different modalities, water quality time-series data feature vectors, fish behavior feature vectors, plant physiological feature vectors, and acoustic data features are each linearly projected through a fully connected layer: E i =W i *X i +b i , where X i W represents a feature of a certain mode. i and b i It is the trainable weights and biases corresponding to the modality, which uniformly map the dimensions of all features from the original dimensions to the preset model dimensions, so that the features of different modalities can be compared and calculated in the same space; For each unified feature vector E i Add a learnable position encoding vector Pi Z i =E i +P i Pi uniquely encodes the modality type of the feature and its position in the time sequence, resulting in the initial embedding sequence Z=[Z1, Z2, ..., Z...]. n ].

[0030] A2. Cross-modal interaction, calculating attention weights; The embedded sequence Z is respectively coupled to three trainable weight matrices W. Q W K W V Multiply them to generate the query matrix Q, the key matrix K, and the value matrix V, where Q = Z * W. Q K=Z*W K V=Z*W V ; Where Q represents the query issued by each feature, wanting to know how relevant other features are to itself; K represents the identity identifier of each feature, used to respond to the query of Q; V represents the substantive information content of each feature, which is the knowledge that is actually transmitted later; The attention score is calculated by multiplying Q and K among all feature pairs: attention score = Q * K. T To prevent gradient vanishing, the scores are divided by the dimension of the K vector. Then, the Softmax function is applied to each row of the scaled score matrix to obtain the attention weight matrix D, specifically: ; D is a calculated attention weight matrix that quantifies the contextual association strength between any two feature vectors in the sequence. Each value in the matrix is ​​D. ij Both represent the degree of attention one feature pays to another in the sequence, D. ij This indicates the weight that should be assigned to the j-th feature information when generating a new representation of the i-th feature.

[0031] A3. Feature weighted fusion, followed by feedforward transformation; By weighting the value vector V with the attention weight matrix D, Z′=D*V. Each vector in the sequence Z′ is no longer isolated information of this modality, but a new feature representation with enhanced context that absorbs all other modality-related information. The above process is executed in parallel across multiple heads, meaning multiple attention calculations are performed in parallel, with each head having its own independent set of W. Q i W K i W V iEach computation can learn to focus on different information in different representation subspaces, and the outputs of all heads are concatenated and linearly transformed to capture complementary information from different representation subspaces; By using a feedforward neural network to perform a nonlinear transformation on the attention-weighted features, the model's ability to fit complex feature interactions is enhanced. Residual connections are first made on the outputs of each sub-layer self-attention layer and feedforward layer, followed by layer normalization to prevent gradient vanishing and model degradation.

[0032] A4. Generate context-aware feature vectors; The Transformer encoder layers consisting of steps A1-A3 are stacked N times. The sequence Z will pass through all these layers in sequence, undergoing N deep interactions and transformations. At the very beginning of the input sequence, a learnable special label [CLS] is pre-inserted. [CLS] is often used in natural language processing to represent the aggregated features of a sequence. Here, it is borrowed to represent the overall state of the system. The final hidden state labeled by this [CLS] aggregates the contextual information of the entire sequence. The vector corresponding to the final layer [CLS] label is the final context-aware feature vector.

[0033] The context-aware feature vector is a fixed-length vector representing the health status of the aquaponics system at the current moment. This vector contains all information from water quality, fish behavior, plant status, and acoustic environment, and this information is dynamically weighted and deeply complementary based on their inherent correlations.

[0034] S4. Through the water quality prediction model construction module, a hybrid water quality prediction model is constructed using a hybrid time-series deep learning model. The model uses the context-aware feature vector to predict key water quality parameters in multiple steps and outputs the prediction results.

[0035] It should be noted that a CNN-LSTM-Transformer hybrid model is adopted, which integrates the advantages of three mainstream neural networks to capture local short-term patterns, long-term dependencies, and global contextual importance, respectively. A context-aware feature vector sequence with a fixed-length historical time window is used as input, and the feature vector at each time step is a comprehensive state representation that integrates multimodal information.

[0036] The forward propagation process of the mixed water quality prediction model is as follows: One-dimensional convolutional neural network (CNN) layers extract local fluctuations and short-term patterns in sequences; Using multiple convolutional kernels of different widths to capture local features at different time scales in parallel, the convolutional operation can identify local features such as the continuous decrease of dissolved oxygen in several consecutive measurements or the instantaneous pulse of ammonia nitrogen after feeding, and output a set of feature maps that preserve the sequence order, but the representation of each point includes information about its neighboring points.

[0037] The bidirectional long short-term memory (LSTM) layer processes the local feature sequences extracted by the CNN and learns long-distance, context-dependent dependencies. It includes a forward LSTM (from past to future) and a backward LSTM (from future to past). When understanding the state at any given time step, the model can simultaneously consider its historical causes and future effects. It outputs the hidden state at each time step, which encodes contextual information based on the entire input sequence.

[0038] The Transformer encoding layer applies global attention weights to the hidden states of all time steps output by the LSTM, enabling the model to learn to focus on predicting the most critical historical moments in the future. The self-attention mechanism dynamically calculates the importance score of each historical time step for the current prediction task; it outputs a feature representation weighted by the global context, highlighting the key event points in the historical sequence that are most indicative of future trends. The output of the Transformer layer is mapped to specific predicted values. One or more fully connected layers are used as decoders. The final output layer node number of the model is determined by the prediction step size multiplied by the number of prediction targets. The output is a vector of predicted values ​​containing various key water quality parameters in the future predetermined time series.

[0039] S5. By embedding an expert rule base and optimization algorithm in the intelligent decision-making module, the optimal collaborative control strategy is generated based on the water quality prediction results and the current system status.

[0040] It should be noted that the intelligent decision-making module adopts a collaborative decision-making mechanism, integrating an expert rule base and optimization algorithms.

[0041] The expert rule base serves as a security foundation, handling explicit, high-priority scenarios. It encapsulates domain knowledge and security principles, existing as production rules in the form of IF-THEN. Each rule is accompanied by a confidence parameter, which can be dynamically adjusted. Among them, security-related rules have the highest priority and can interrupt other operations.

[0042] When faced with multiple control objectives, limited resources, or potential conflicts between strategies, simple rules cannot make the optimal judgment. The decision-making problem is modeled as a multi-objective optimization problem, and an optimization algorithm is used for decision-making. The optimization algorithm adopts a hybrid strategy, integrating fuzzy logic and reinforcement learning to cope with different scenarios. The fuzzy logic is used to handle empirical regulation of uncertainty. It transforms fuzzy language into specific control quantities, defines inputs, performs inference through a fuzzy rule base, and finally defuzzifies to obtain precise control instructions. The reinforcement learning described is used to learn long-term optimal policies in complex and dynamic environments. The agent learns a policy function that maximizes long-term cumulative rewards through continuous interaction with the aquaponics system, specifically including: Status: Current system status, including prediction information; Actions: Combinations of control actions for all executable devices, such as oxygen pump power, feeding amount, and water pump frequency; Reward function: A mathematical function used to evaluate the quality of a strategy, in the form of: Reward = w1*(water quality stability) - w2*(total energy consumption) + w3*(equipment wear and tear cost) - w4*(strategy oscillation penalty), where w1, w2, ... are weights, representing the relative importance of different objectives.

[0043] The rule base and optimization algorithm ultimately output multiple preliminary policies that may conflict. The policy arbitrator makes the final decision based on priority, security, resource constraints, and synergy between policies, and outputs a structured, machine-readable set of collaborative control policies.

[0044] S6. Receive the optimal coordinated control strategy through the device control module, convert it into control instructions for the specific execution device, and drive the execution device to change the system state after verification.

[0045] It should be noted that the device control module maintains a device driver library and a device digital twin system registry, which records detailed information about each physical actuator, including: Device identifier: unique ID and physical address; Equipment capabilities: Supported operations, measurement range, accuracy, and response time; Communication protocol: The specific communication protocol used by the device.

[0046] The specific steps for the drive execution device to change the system state are as follows: B1. Perform security checks, logical interlocks, and state consistency checks; specifically: Boundary verification: Checks whether the command parameters are within the safe operating range of the equipment. For example, if the command requires the pH adjustment pump dosage to be set to 150%, the system will intercept the command and trigger an alarm because it exceeds the equipment's capacity limit. Logical interlocks: Check whether the instruction to be executed will conflict with the instruction currently being executed; For example, the rule stipulates that "fine-tuning of pH is prohibited when performing large-volume water replacement", because drastic changes in water flow will cause pH adjustment to fail and waste reagents. The control module will delay the pH adjustment command until the water replacement operation is completed. Status consistency check: Query the current actual status of the device; for example, before executing the "turn on heater" command, confirm that the heater is indeed in an available offline state, rather than a faulty state.

[0047] B2. Perform instruction scheduling and queue management; When multiple strategies need to be executed simultaneously, the control module establishes an execution queue based on the priority and timing dependencies of the strategies, managing the execution sequence of instructions, especially for operations with strict sequential requirements. For example, "start the circulating water pump first, then start the aerator pump after a 10-second delay," to prevent the aerator pump from running unloaded.

[0048] B3. Drive the execution device to change the system state; The final low-level instructions are sent to the corresponding execution device through the corresponding industrial communication protocol, and the feedback signal of the device is read to confirm that the instructions have been successfully executed. After the instructions are issued, the control module immediately updates the status of the internal device digital twin.

[0049] It should be noted that if no confirmation is received from the device within a specified time after the command is issued, a limited number of retries will be performed. If the retries still fail or communication is interrupted, an abnormal alarm will be sent immediately, and the device will be marked as offline or faulty.

[0050] S7. Through the performance evaluation module, during the set observation period after the execution of the equipment, the deviation between the actual change and the expected change of the target variable is analyzed to calculate the control effect coefficient and generate an evaluation report.

[0051] It should be specifically noted that the calculation of the regulation effect coefficient is as follows: Effect control coefficient η = (actual improvement value of target variable - expected natural improvement value of target variable) / instruction intensity; Where the actual improvement value of the target variable = t0 + T actual value - t0 actual value, that is, the actual change of the target variable from the instruction execution time t0 to the observation end time t0 + T within the preset observation period T; Where the expected natural improvement value of the target variable = t0 + T predicted value - t0 actual value refers to the amount of natural change that the system prediction model believes the target variable will undergo within the same observation period T under the assumption that no control command is executed. At time t0, with the current system state as input, the water quality prediction model is called to predict the target variable value at time T in the future under the assumption that no current control command is executed, which is the t0 + T predicted value. The instruction strength = (instruction setting value - instruction setting lower limit) / (instruction setting upper limit - instruction setting lower limit) normalizes the physical output of the control instruction, making the instructions of different types and magnitudes comparable. For switching quantities or discrete actions, it can be defined as 1, or a weight representing its strength can be used.

[0052] It should be explained that when η > 0, the regulation is effective, and the larger the positive value, the greater the additional improvement brought by the unit instruction intensity, and the more significant the effect. When η≈0, the control is ineffective, the actual improvement is no different from the natural trend of the system, and the command does not produce the expected effect. When η < 0, regulation has the opposite effect; the execution of instructions worsens the situation, falling short of the natural evolutionary trend.

[0053] Generate an effectiveness evaluation report for each important instruction, including: effectiveness coefficient, whether the expected results were achieved, and potential interfering factors.

[0054] S8. Receive feedback data from the execution effect evaluation module through the online learning and optimization module, adaptively optimize the decision-making strategy through reinforcement learning algorithm, trigger online fine-tuning of the water quality prediction model, and dynamically evolve the expert rule base.

[0055] It should be specifically noted that the adaptive optimization of the decision-making strategy through reinforcement learning algorithm specifically refers to: Construct a reinforcement learning framework. State: defined as the context-aware feature vector output by the multimodal feature fusion module, which contains comprehensive information about the system state. Action: Defined as the set of all control instructions that the intelligent decision-making module can execute; Reward: Defined as the control effect coefficient calculated by the performance evaluation module, it directly and quantitatively reflects the quality of the action; Environment: The entire aquaponics physical system and its digital twin.

[0056] The PPO algorithm is optimized using a proximal policy. Each decision execution generates an experience tuple (state St, action At, reward Rt, new state St+1) and stores it in an experience replay buffer. Periodically, a batch of experience data is sampled from the buffer. The PPO algorithm uses this data to update its Actor network (responsible for decision-making) and Critic network (responsible for evaluating state value) through gradient ascent. The goal of the update is to maximize the future cumulative expected reward. The updated, better-performing policy network will gradually replace the old policy in the intelligent decision-making module. Through learning, the system will gradually learn which coordinated control actions can obtain the highest long-term reward under what complex states.

[0057] The online fine-tuning of the water quality prediction model specifically involves: When the system detects that the prediction error continues to increase, or when the accumulated new "state-action-result" data reaches a certain scale, it automatically triggers the fine-tuning process, inputs the historically accumulated context-aware feature vectors, and outputs the corresponding verified actual water quality data. This new data contains the system's real response after performing various control actions, which enables the model to learn the impact of intervention on the system dynamics, thereby making more accurate predictions in the future. Using the mixed water quality prediction model as the initial weights, the model is trained for additional rounds using a new dataset and a very small learning rate.

[0058] The dynamic evolution of the expert rule base is specifically as follows: A confidence parameter is maintained for each rule in the rule base. When a rule is triggered and executed, its confidence is adjusted according to the effect adjustment coefficient η. When η is consistently positive and greater than a preset first threshold, the confidence is increased; when η is consistently negative or zero, the confidence is decreased; when the confidence of a rule is lower than a preset second threshold, the system can automatically disable it and notify the administrator. Efficient strategies discovered by reinforcement learning can be abstracted and solidified into new expert rules and added to the rule base, enabling the expert system to grow together with AI.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that the various embodiments of this application can be implemented by means of software or software combined with necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware functions. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a computer device, such as including but not limited to a personal computer, server, or network device, to execute all or part of the steps of the method described in any embodiment of this application.

[0060] The foregoing has described exemplary embodiments of this application. It should be understood that the above exemplary embodiments are not restrictive but illustrative, and the scope of protection of this application is not limited thereto. It should be understood that those skilled in the art can make modifications and variations to the embodiments of this application without departing from the spirit and scope of this application, and such modifications and variations should be within the scope of protection of this application.

Claims

1. A water quality prediction and control system for aquaponics based on multimodal data fusion, characterized in that, Specifically, it includes: Multimodal data sensing module: used to deploy sensor arrays to collect multimodal data of the aquaponics system in real time, including physicochemical modal data, optical modal data and acoustic modal data; Feature extraction module: Used to process the collected multimodal data and extract features from the processed data to obtain multimodal feature sequences, including water quality time series data features, fish behavior features, plant physiological features and acoustic data features; Multimodal feature fusion module: Used to deeply fuse the multimodal feature sequences using an attention-based model to construct a context-aware feature vector that can comprehensively describe the health status of the aquaponics system; Water quality prediction model construction module: used to construct a hybrid water quality prediction model using a hybrid time-series deep learning model, perform multi-step prediction of key water quality parameters through the context-aware feature vector, and output the prediction results; Intelligent decision-making module: This module incorporates an expert rule base and optimization algorithms to generate the optimal collaborative control strategy based on the water quality prediction results and the current system status. Equipment control module: Used to receive the optimal coordinated control strategy, convert it into control instructions for specific execution devices, and drive the execution devices to change the system state after verification; Implementation effect evaluation module: Used to calculate the control effect coefficient and generate an evaluation report by analyzing the deviation between the actual change and the expected change of the target variable during the set observation period after the execution of the equipment; Online learning and optimization module: It is used to receive feedback data from the performance evaluation module, adaptively optimize decision-making strategies through reinforcement learning algorithms, trigger online fine-tuning of the water quality prediction model, and dynamically evolve the expert rule base. Visual interaction module: used to provide a global system status dashboard, historical data query, remote manual control of equipment, and early warning information push functions.

2. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The multimodal data sensing module specifically includes: Water quality sensor array: Deploy immersion or flow-through sensors to monitor water bodies in real time and collect physicochemical modal data, including pH value, dissolved oxygen concentration, temperature, conductivity and redox potential of the water body; Optical acquisition unit: This unit comprises waterproof cameras deployed underwater in the fishpond, fixed-point cameras targeting plant roots, and wide-angle cameras monitoring the fishpond surface. It captures high-definition video streams and still images, collecting optical modal data, specifically including: underwater images: used to analyze fish behavior, identifying activity levels, feeding desires, and whether fish are surfacing; surface images: used to analyze fish feeding patterns and determine if there is uneaten food; and plant images: used to analyze plant leaf color and morphology through image recognition to assess health status. Acoustic acquisition unit: A high-frequency underwater microphone deployed underwater to collect acoustic modal data including fish activity, feeding, and equipment operation.

3. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The water quality time series data features specifically include: statistical features: calculating the mean, variance, and trend slope within the sliding window; periodic features: extracting the diurnal periodic pattern and identifying abnormal deviations; and event-triggered features: marking feeding as an important event and calculating the rate of change of water quality parameters before and after feeding. The fish behavior characteristics specifically include: identifying fish from the video stream based on a target detection model, and calculating: average swimming speed; group aggregation degree: calculating the center point of the fish group and analyzing the variance of the distance between the individual and the center; and surface activity frequency: counting the number of times the fish mouths touch the water surface per unit time. The plant physiological characteristics specifically include: color space analysis: plant leaf color histogram based on HSV color space, used to detect nutrient deficiency symptoms such as yellowing and whitening; texture characteristics: leaf texture based on local binary mode or gray-level co-occurrence matrix, used to determine leaf dehydration or disease; morphological characteristics: morphological characteristics based on leaf spread and contour curvature, used to quantify whether the leaves are wilting. The acoustic data features specifically include: Mel frequency cepstral coefficients: used to extract the activity level of fish; spectral centroid and spectral roll-off point: describing the brightness and energy distribution of the sound, used to distinguish between normal swimming sounds and rapid tail-wagging sounds when startled; and sound pressure level temporal variation: analyzing the variation pattern of sound pressure level in specific frequency bands to identify abnormally high-pitched sounds.

4. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The specific steps for constructing the context-aware feature vector are as follows: A1. Input embedding and positional encoding; Input feature sequences from different modalities are linearly projected through a fully connected layer: E i =W i *X i +b i , where X i W represents a feature of a certain mode. i and b i These are the trainable weights and biases corresponding to this modality, which unify the dimensions of all features from their original dimensions to the preset model dimensions; this is the unified feature vector E. i Add a learnable position encoding vector P i Z i =E i +P i , where P i The modality type of the feature and its position in the time series are uniquely encoded, resulting in the initial embedding sequence Z=[Z1, Z2, ..., Z...]. n ]; A2. Cross-modal interaction, calculate attention weights; interpolate the embedded sequence Z with three trainable weight matrices W. Q W K W V Multiply them to generate the query matrix Q, the key matrix K, and the value matrix V, where Q = Z * W. Q K=Z*W K V=Z*W V Where Q represents the query issued by each feature, wanting to know the relevance of itself to other features; K represents the identity of each feature, used to respond to the query in Q; V represents the substantive information content of each feature, which is the knowledge that is actually transmitted later; the attention score Q*K among all features is calculated by the dot product of Q and K. T Divide the scores by the dimension of the K vectors and apply the Softmax function to each row of the scaled score matrix to obtain the attention weight matrix D, specifically: ; D is a calculated attention weight matrix that quantifies the contextual association strength between any two feature vectors in the sequence. Each value in the matrix is ​​D. ij Both represent the degree of attention one feature pays to another in the sequence, D. ij This indicates the weight that should be assigned to the j-th feature information when generating a new representation of the i-th feature; A3. Feature weighted fusion and feedforward transformation: The value vector V is weighted and summed through the attention weight matrix D, Z′=D*V. Each vector in the sequence Z′ absorbs new context-enhanced feature representations that are relevant to all other modalities. The above process is executed in parallel on multiple heads. Each computation can learn to focus on different information in different representation subspaces. The outputs of all heads are concatenated and linearly transformed. A feedforward neural network is used to perform nonlinear transformation on the attention-weighted features. Residual connections are first made on the outputs of the self-attention layer and the feedforward layer of each sub-layer, and then layer normalization is performed. A4. Generate context-aware feature vectors; The Transformer encoder layers consisting of steps A1-A3 are stacked N times. The sequence Z will pass through all these layers in sequence, undergoing N deep interactions and transformations. At the very beginning of the input sequence, a learnable special label [CLS] is pre-inserted. The final hidden state labeled by this [CLS] aggregates the context information of the entire sequence. The vector corresponding to the final layer [CLS] label is the final context-aware feature vector.

5. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The forward propagation process of the mixed water quality prediction model is as follows: One-dimensional convolutional neural network (CNN) layers extract local fluctuations and short-term patterns in sequences; multiple convolutional kernels of different widths are used to capture local features at different time scales in parallel, outputting a set of feature maps that preserve the sequence order, but the representation of each point includes information about its neighboring points; The bidirectional Long Short-Term Memory (LSTM) layer processes the local feature sequences extracted by CNN and learns long-distance, context-dependent dependencies. It includes a forward LSTM (from past to future) and a backward LSTM (from future to past), so that the model can consider both its historical causes and future effects when understanding the state at any given time step. It outputs the hidden state at each time step, which encodes contextual information based on the entire input sequence. The Transformer encoding layer applies global attention weights to the hidden states of all time steps in the LSTM output, focusing on predicting the most critical historical moments in the future. The self-attention mechanism dynamically calculates the importance score of each historical time step for the current prediction task; outputs a feature representation weighted by the global context; maps the output of the Transformer layer to specific prediction values, and outputs a prediction vector containing various key water quality parameters for the future predetermined time series.

6. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The intelligent decision-making module adopts a collaborative decision-making mechanism, integrating an expert rule base and optimization algorithms; The expert rule base serves as a security foundation, handling explicit, high-priority scenarios. It encapsulates domain knowledge and security principles, existing as production rules in the form of IF-THEN. Each rule is accompanied by a confidence parameter, which can be dynamically adjusted. Among them, security-related rules have the highest priority and can interrupt other operations. When faced with multiple control objectives, limited resources, or potential conflicts between strategies, simple rules cannot make the optimal judgment. Therefore, the decision-making problem is modeled as a multi-objective optimization problem, and optimization algorithms are used to make the decision. The optimization algorithm employs a hybrid strategy, integrating fuzzy logic and reinforcement learning to address different scenarios. The fuzzy logic is used to handle empirical regulation of uncertainty, transforming fuzzy language into specific control quantities, defining inputs, performing inference through a fuzzy rule base, and finally defuzzifying to obtain precise control instructions. The reinforcement learning is used to learn long-term optimal policies in complex and dynamic environments. The agent learns a policy function that maximizes long-term cumulative rewards through continuous interaction with the aquaponics system. The rule base and optimization algorithm ultimately output multiple preliminary policies that may conflict. The policy arbitrator makes the final decision based on priority, security, resource constraints, and synergy between policies, and outputs a structured, machine-readable set of collaborative control policies.

7. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The specific steps for the drive execution device to change the system state are as follows: B1. Perform security checks, logical interlocks, and state consistency checks; specifically: boundary checks: check whether the instruction parameters are within the safe operating range of the device; logical interlocks: check whether the instruction to be executed will conflict with the instruction currently being executed; state consistency checks: query the current actual state of the device. B2. Perform instruction scheduling and queue management; When multiple strategies need to be executed simultaneously, the control module establishes an execution queue based on the priority and timing dependencies of the strategies to manage the execution order of instructions, especially for operations with strict sequential requirements. B3. Drive the execution device to change the system state; send the final low-level instruction to the corresponding execution device through the corresponding industrial communication protocol, and read the device's feedback signal to confirm that the instruction has been successfully executed. After the instruction is issued, the control module immediately updates the state of the internal device digital twin.

8. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The calculation of the regulation effect coefficient is as follows: Effect control coefficient η = (actual improvement value of target variable - expected natural improvement value of target variable) / instruction intensity; Where the actual improvement value of the target variable = t0 + T actual value - t0 actual value, that is, the actual change of the target variable from the instruction execution time t0 to the observation end time t0 + T within the preset observation period T; Where the expected natural improvement value of the target variable = t0 + T predicted value - t0 actual value refers to the amount of natural change that the system prediction model believes the target variable will undergo within the same observation period T under the assumption that no control command is executed. At time t0, with the current system state as input, the water quality prediction model is called to predict the target variable value at time T in the future under the assumption that no current control command is executed, which is the t0 + T predicted value. Wherein, instruction strength = (instruction setting value - instruction setting lower limit) / (instruction setting upper limit - instruction setting lower limit); When η > 0, the regulation is effective, and the larger the positive value, the greater the additional improvement brought by the unit instruction intensity, and the more significant the effect. When η≈0, the control is ineffective, and the actual improvement is no different from the natural trend of the system, and the command does not produce the expected effect; when η<0, the control has the opposite effect, and the execution of the command makes the situation worse, which is lower than the natural evolution trend.

9. The aquaponics water quality prediction and control system based on multimodal data fusion according to claim 1, characterized in that: The global status dashboard specifically includes: Core KPI indicator cards: clearly displaying the most critical real-time data in the form of large numbers, dashboards, or trend arrows, and intuitively indicating whether the status is normal through color coding (green / yellow / red); Multimodal data fusion visualization: Water quality time series curves: displaying the predicted water quality curve and the actual measured curve on the same chart; Biological behavior indices: transforming abstract features such as fish activity, surfacing frequency, and plant health index into visualized indicator bars or trend charts; Equipment status panel: displaying the real-time status of all executing equipment in the form of diagrams or lists, and integrating a one-click manual control button; Transparent AI decision-making display: displaying the currently effective control strategies, informing users what the system is doing and why; Displaying the control effect evaluation results, showing the effect coefficient of recent control actions in the form of notifications or reports; The historical data query specifically includes: multi-dimensional query: users can freely select the time range and data type, and support comparative analysis of different data series; drill-down analysis: supports drilling down from the macro trend chart to detailed data at a specific point in time and the on-site snapshot at that time; data export: provides the function of exporting raw data for users to perform offline in-depth analysis or generate reports; The remote manual control of the device specifically includes: a manual mechanism: users can directly and manually control any device through the control panel at any time. When a manual command is issued, the system will automatically pause the relevant automatic control strategy and prompt the user to restore the automatic mode after the manual operation is completed; a strategy intervention mechanism: users can review, modify or reject the AI-generated strategy. The aforementioned early warning information push function specifically includes: a multi-level early warning system: alerts (non-urgent information); warnings (situations requiring attention); and alarms (emergencies requiring immediate intervention). Intelligent push channels: The system automatically selects push channels based on the early warning level and time period. Warnings and alerts are displayed in the web interface notification center; alarms are delivered directly to users via mobile app push, SMS, or telephone. Closed-loop early warning management: The system tracks the status of each early warning, requiring user confirmation before it can be marked as processed.

Images