Marine pasture disaster decision method based on reinforcement learning

Abstract

The application discloses a kind of marine ranching disaster decision-making methods based on reinforcement learning.Method includes: interactive environment module constructs the virtual pasture sea area of marine ranching;Disaster judgment module judges whether marine ranching occurs disaster, and through action space module, preliminary post-disaster action is taken to virtual pasture sea area Output feedback result;Decision module outputs preliminary decision data;Disaster judgment module judges whether marine ranching disaster ends and outputs judgment result;Reward update module calculates reward value;After correction, in turn input parameter optimization module and decision module update optimization;Repeatedly obtain trained disaster judgment module and decision module;Disaster judgment module judges that marine ranching occurs disaster, and trained decision module outputs monitoring decision data, and according to monitoring decision data, marine ranching occurs disaster is decided.The application can improve the accuracy and flexibility of marine ranching disaster decision-making, solve the problem of backward risk disaster decision-making technology, and improve the control efficiency.

Description

Technical Field

[0001] This invention relates to a disaster decision-making method for marine ranches, specifically a disaster decision-making method for marine ranches based on reinforcement learning. Background Technology

[0002] In the field of marine environmental disaster decision-making research, the Analytic Hierarchy Process (AHP), a multi-objective decision analysis method integrating quantitative and qualitative analysis, is widely used. Its principle is to divide the problem into hierarchical levels, classify and decompose relevant factors to form a multi-level structural model, and assign values ​​to factors layer by layer. The AHP simplifies complex problems by decomposing the research problem into hierarchical and quantifiable components, thus making the analysis and processing of the problem easier. However, when dealing with disaster decision-making scenarios related to marine ranching in complex sea conditions, the AHP becomes quite limited due to the existence of physical factors and patterns in the marine environment that cannot be accurately quantified and stratified.

[0003] A major research goal in the field of artificial intelligence is to achieve fully autonomous intelligent agents. These agents are capable of interacting with their environment, learning optimal behavior based on environmental feedback, and continuously improving their action strategies through repeated experimentation. Deep Reinforcement Learning (DRL) provides the theoretical foundation for achieving this goal. As an important branch of artificial intelligence research, it is considered key to realizing human-like intelligence and has received widespread attention from academia and industry.

[0004] DRL is an end-to-end perception and control system with strong versatility. Its learning process can be described as follows: at each time step, the agent interacts with the environment to obtain a high-dimensional observation, and uses DL methods to perceive the observation to obtain specific state feature representations; the value function of each action is evaluated based on the expected reward, and the current state is mapped to the corresponding action through a certain strategy; the environment reacts to this action and obtains the next observation.

[0005] By continuously iterating the above process, the optimal strategy for achieving the goal can eventually be obtained. On the one hand, DRL has a powerful ability to represent policies and states, and can be used to simulate complex decision-making processes; on the other hand, reinforcement learning endows agents with self-supervised learning capabilities, enabling them to interact autonomously with the environment and continuously improve through trial and error. However, DRL has not yet been applied in the construction of marine ranches. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention provides a marine ranching disaster decision-making method based on reinforcement learning. This invention is a marine ranching disaster decision-making algorithm based on reinforcement learning, which solves the shortcomings of existing technologies such as low efficiency, lack of flexibility, and weak linkage in dynamic decision-making and planning for marine disasters involving marine ranches.

[0007] The technical solution adopted in this invention is:

[0008] The marine ranching disaster decision-making method of the present invention includes the following steps:

[0009] Step 1: Obtain historical ranch status data of the marine ranch before the current time. Input the historical ranch status data into the data processing module for data preprocessing to obtain historical ranch preprocessed status data. Input the historical ranch preprocessed status data into the interactive environment module. In the interactive environment module, construct a virtual ranch sea area of ​​the marine ranch. That is, use the preprocessed historical ranch data as input to construct a marine ecological simulation and evaluation model based on artificial neural networks, which serves as the main body of the interactive environment module.

[0010] Step 2: Input the preprocessed historical ranch status data into the disaster judgment module. The disaster judgment module determines whether a disaster has occurred in the marine ranch. When a disaster occurs in the marine ranch, a preset post-disaster action is input into the interactive environment module through the action space module to take the preset post-disaster action on the virtual ranch sea area. The interactive environment module outputs the feedback result generated by the virtual ranch sea area. The obtained ranch status data are all lagging data, that is, the historical ranch data obtained at the moment before the current moment is actually the historical ranch data N hours before the current moment of the marine ranch. That is, when it is judged that a disaster has occurred in the marine ranch based on the historical ranch data at the moment before the current moment, the actual marine ranch has experienced a marine disaster N hours ago.

[0011] Step 3: Obtain real-time ranch status data of the marine ranch. Input the real-time ranch status data into the data processing module for data preprocessing to obtain real-time ranch preprocessed status data, and input it into the decision module. The decision module outputs preliminary decision data.

[0012] Step 4: Input the preliminary decision data into the interactive environment module. The interactive environment module outputs the predicted state value and state change of the virtual ranch sea area. Input the historical ranch state preprocessing data, the feedback results generated by the virtual ranch sea area, the predicted state value and state change into the disaster judgment module. The disaster judgment module judges whether the disaster of the marine ranch has ended and outputs the judgment result.

[0013] In practice, the disaster assessment module combines historical ranch status preprocessing data, feedback results from virtual ranch sea areas, predicted status values ​​and status changes with early warning conditions and thresholds to summarize a parameter-induced disaster correlation formula. This formula is used to determine whether the current ranch sea area and virtual sea area environment are in a risk disaster state, i.e., whether the disaster has ended. Specifically, it can determine whether the predicted status value is still within the risk range.

[0014] Step 5: Input the judgment result output by the disaster judgment module, the predicted state value of the virtual ranch sea area, and the state change into the reward update module. The reward update module calculates the reward value for this time.

[0015] Step Six: Correct the judgment results output by the disaster judgment module and the predicted state value of the virtual ranch sea area based on the real-time ranch preprocessing state data; process the corrected judgment results and predicted state values, preliminary decision data, marine ranch state changes and environmental prediction error input parameter optimization module, and then input the processed output into the decision module for update and optimization.

[0016] During the correction process, the predicted state value of the virtual ranch sea area is corrected to the real-time ranch preprocessed state data. At the same time, the real-time state of the marine ranch is determined, and the judgment result output by the disaster judgment module is corrected to the real-time state of the marine ranch.

[0017] Step 7: Repeat steps 1 to 6 to train the disaster judgment module and decision-making module repeatedly until the reward value calculated by the reward update module converges to the maximum value. Stop training the disaster judgment module and decision-making module to obtain the trained disaster judgment module and decision-making module. Based on parameter optimization, it is also necessary to reduce the number of training rounds required.

[0018] Step 8: Real-time acquisition of marine ranch monitoring status data and input into the data processing module for data preprocessing to obtain ranch preprocessed monitoring status data. Input the ranch preprocessed monitoring status data into the trained disaster judgment module. When the disaster judgment module determines that a disaster has occurred in the marine ranch, input the ranch preprocessed monitoring status data into the trained decision module, process it, and output monitoring decision data. Make decisions for the marine ranch that has been affected by the disaster based on the monitoring decision data.

[0019] The historical and real-time ranch status data of the marine ranch include multi-parameter sensor data, turbidity sensor data, flow velocity data, and ecological simulation and forecast data.

[0020] Marine multi-parameter sensor data includes serial number, date, time, conductivity, chlorophyll, pH, dissolved oxygen, and sound velocity; turbidity sensor data includes turbidity data; current velocity data includes layer number, depth, raw current velocity data, current velocity in the x-direction, current velocity in the y-direction, current velocity in the z-direction, composite current velocity, and composite current velocity direction; ecological simulation and forecast data includes time, longitude, latitude, depth, water level, salinity, water temperature, eastward current velocity, and northward current velocity.

[0021] Historical pasture status data is input into the data processing module for preprocessing to obtain historical pasture preprocessed status data. Specifically, marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data from the historical pasture status data are input into the data processing module for processing such as missing value supplementation, random sampling, and serialization. In practice, missing values ​​are supplemented for any missing data. Each set of pasture status data also needs to be compressed according to the size of the dataset. The size of the dataset is mainly determined by the depth value and the number of data entries. Small datasets are processed using vertical averaging, while large datasets are processed using a VAE model for data compression. The processed outputs are collectively used to construct the historical pasture preprocessed status data.

[0022] In step one, the preprocessed historical ranch status data is input into the interactive environment module. The interactive environment module then constructs a virtual ranch area for the marine ranch. Specifically, the interactive environment module constructs the virtual ranch area based on the preprocessed historical ranch status data, the deployment layout of various devices within the marine ranch, and the two-dimensional shallow water equations and embedded second-order moment turbulent closure sub-model of the marine ranch area. This virtual ranch area can predict the environmental data for the next decision-making time based on different environmental data.

[0023] In step two, the disasters that occur in the marine ranch specifically include meteorological disasters, hydrological disasters, and geological disasters. The historical ranch status preprocessing data is input into the disaster judgment module, which determines whether the marine ranch has experienced a disaster. Specifically, the disaster judgment module determines whether the marine ranch meets the early warning conditions for meteorological disasters, hydrological disasters, or geological disasters based on the historical ranch status preprocessing data. If the conditions are met, the disaster judgment module determines that the marine ranch is in a state of meteorological disaster, hydrological disaster, or geological disaster.

[0024] Meteorological disasters, hydrological disasters, and geological disasters all have specific early warning conditions, namely the early warning interval standards corresponding to national standards. For example, storm surges, which belong to hydrological disasters, are warned when the preprocessed historical pasture data shows the above-warning tide level, wind speed, and one-third of the tide height H. 1 / 3 If all conditions exceed the warning criteria, the marine ranch is considered to be in the event of a storm surge disaster.

[0025] In step two, the action space module includes several preset post-disaster actions. Each preset post-disaster action corresponds to a meta-action taken when a parameter value exceeds the warning value. The parameter value exceeding the warning value is one of the parameter values ​​in the historical ranch status data of the marine ranch, that is, one of the parameter values ​​contained in the marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data, such as wind speed and tide level. The preset post-disaster actions include the start and stop time, start and stop duration, movement direction, and movement speed of the equipment measuring the parameter value exceeding the warning value.

[0026] Preset post-disaster actions must be operations supported by IoT devices or data acquisition devices. Preset post-disaster actions also need to account for the lag in device operation, which reduces the overall efficiency of the actions. When multiple parameters are abnormal, it is also determined to be a certain type of disaster. For each decision step, only one decision action is selected from the action space. This action has a certain probability p that it is random (exploratory nature), and the remaining probability 1-p is to select the action with the highest current reward. A decision round consists of an action sequence composed of all actions from one or more decision steps, which can also be called a decision scheme.

[0027] The feedback results generated by the interactive environment module are specifically the ranch status data after the virtual ranch sea area takes preset post-disaster actions.

[0028] In step three, the decision module is a Deep Q-Network (DQN), specifically employing a dual-memory model LSTM. The LSTM model comprises a short-term memory network and a long-term memory network connected sequentially. The short-term memory network consists of two components: a Deep Q-Network for learning the current task and an experience replay network containing only data from the current task. The long-term memory network also comprises two components: a Deep Q-Network containing knowledge learned from all tasks since the beginning, and a generative adversarial network (GAN) for generating representations of these reinforcement learning task experiences. The decision module is constructed based on the DQN (Deep Q-learning Network) algorithm of deep learning, trained offline using an off-policy strategy, with value function approximation to the neural network construction, and employing a target network and experience replay methods for network training.

[0029] In step four, the preliminary decision data specifically consists of an action sequence composed of one or more preset post-disaster actions in the action space module. The preliminary decision data is input into the interactive environment module. After the interactive environment module takes the action sequence, it outputs the predicted state value and state change of the virtual ranch sea area. The predicted state value of the virtual ranch sea area is specifically the ranch state data after the virtual ranch sea area takes the action sequence. The state change of the virtual ranch sea area is the change in the ranch state data before and after taking the action sequence.

[0030] In actual processing, the interactive environment module can only receive sea area status parameters as input. Therefore, it is necessary to convert the action sequence in the preliminary decision data into an increase or decrease sequence of parameter values ​​exceeding the warning value in the pasture sea area status data of the previous decision stage before inputting it into the interactive environment module. At the same time, it is necessary to consider the differences in the parameter change rate corresponding to the action in different disaster scenarios.

[0031] When a marine ranch experiences a meteorological, hydrological, or geological disaster, and several parameter values ​​in the real-time ranch status data exceed the warning value (i.e., several parameter values ​​in the (micro)delayed data exceed the warning value), then pre-set post-disaster actions need to be taken to regulate the parameters exceeding the warning value, thus forming preliminary decision data. For example, in a storm surge disaster scenario, the parameters exceeding the warning value are wind speed, tide level, wave height, and current velocity. At this time, the pre-set post-disaster actions include the movement direction and speed of equipment for measuring wind speed, tide level, wave height, and current velocity. These actions together constitute preliminary decision data.

[0032] In step four, the disaster judgment module determines whether the disaster in the marine ranch has ended and outputs the judgment result. When all the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value do not exceed the warning value, the disaster in the marine ranch is judged to have ended. When one or more of the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value still exceed the warning value, the disaster in the marine ranch is judged to have not ended.

[0033] In step five, the judgment result output by the disaster judgment module is input into the reward update module. The reward update module calculates the reward value for that time. Each judgment result output by the disaster judgment module represents one decision step time. When the disaster judgment module determines that the disaster in the marine ranch has not ended, it provides a negative feedback value based on the current decision step time. When the disaster judgment module determines that the disaster in the marine ranch has ended, it provides a positive feedback value based on the disaster type. When the total decision step time consumed during training exceeds the response time, the training is considered to have ended, and the decision model continues to be trained on the disaster data until the model can eliminate / avoid / reduce the risk of disaster within the response time.

[0034] The state change of the marine ranch is specifically the change between the real-time ranch preprocessed state data and the real-time ranch preprocessed state data after the interactive environment module takes the action sequence; the environmental prediction error of the marine ranch is specifically the error between the predicted state value of the virtual ranch sea area and the real-time ranch preprocessed state data after the interactive environment module takes the action sequence.

[0035] The beneficial effects of this invention are:

[0036] This invention can increase the standard decision-making dataset for marine disasters in marine ranches, improve the accuracy and flexibility of ranch disaster decision-making, solve the problem of lagging marine ranch risk disaster decision-making technology, and increase the per capita marine area under management and management efficiency. It can be combined with the corresponding ranch decision-making system to realize functions such as auxiliary decision-making under human supervision, provision of the best decision-making scheme, and autonomous decision-making in the absence of human response. Attached Figure Description

[0037] Figure 1 A schematic diagram of a ranch disaster decision-making model;

[0038] Figure 2 The diagram shows the architecture of the quadruple set;

[0039] Figure 3 Implement the structure diagram for the interactive environment module;

[0040] Figure 4 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The marine ranching disaster decision-making method of the present invention includes the following steps:

[0043] Step 1: Obtain historical ranch status data of the marine ranch before the current time. Input the historical ranch status data into the data processing module for data preprocessing to obtain historical ranch preprocessed status data. Input the historical ranch preprocessed status data into the interactive environment module. In the interactive environment module, construct a virtual ranch sea area of ​​the marine ranch. That is, use the preprocessed historical ranch data as input to construct a marine ecological simulation and evaluation model based on artificial neural networks, which serves as the main body of the interactive environment module.

[0044] Both historical and real-time marine ranch status data include multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation and forecast data. Multi-parameter sensor data includes serial number, date, time, conductivity, chlorophyll content, pH, dissolved oxygen, and sound velocity; turbidity sensor data includes turbidity data; current velocity data includes layer number, depth, raw current velocity data, x-direction current velocity, y-direction current velocity, z-direction current velocity, composite current velocity, and composite current velocity direction; ecological simulation and forecast data includes time, longitude, latitude, depth, water level, salinity, water temperature, eastward current velocity, and northward current velocity.

[0045] Historical pasture status data is input into the data processing module for preprocessing to obtain historical pasture preprocessed status data. Specifically, marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data from the historical pasture status data are input into the data processing module for processing such as missing value supplementation, random sampling, and serialization. In practice, missing values ​​are supplemented for any missing data. Each set of pasture status data also needs to be compressed according to the size of the dataset. The size of the dataset is mainly determined by the depth value and the number of data entries. Small datasets are processed using vertical averaging, while large datasets are processed using a VAE model for data compression. The processed outputs are collectively used to construct the historical pasture preprocessed status data.

[0046] In step one, the preprocessed historical ranch status data is input into the interactive environment module. This module constructs a virtual ranch area for the marine ranch. Specifically, the interactive environment module constructs the virtual ranch area based on the preprocessed historical ranch status data, the deployment layout of various devices within the marine ranch, and the two-dimensional shallow water equations and embedded second-order moment turbulent closure sub-model of the marine ranch's location. The virtual ranch area can predict the environmental data for the next decision-making time based on different environmental data.

[0047] Step 2: Input the preprocessed historical ranch status data into the disaster judgment module. The disaster judgment module determines whether a disaster has occurred in the marine ranch. When a disaster occurs in the marine ranch, a preset post-disaster action is input into the interactive environment module through the action space module to take the preset post-disaster action on the virtual ranch sea area. The interactive environment module outputs the feedback result generated by the virtual ranch sea area. The obtained ranch status data are all lagging data, that is, the historical ranch data obtained at the moment before the current moment is actually the historical ranch data N hours before the current moment of the marine ranch. That is, when it is judged that a disaster has occurred in the marine ranch based on the historical ranch data at the moment before the current moment, the actual marine ranch has experienced a marine disaster N hours ago.

[0048] In step two, the disasters that occur in the marine ranch specifically include meteorological disasters, hydrological disasters, and geological disasters. The historical ranch status preprocessing data is input into the disaster judgment module. The disaster judgment module determines whether the marine ranch has experienced a disaster. Specifically, the disaster judgment module determines whether the marine ranch meets the early warning conditions for meteorological disasters, hydrological disasters, or geological disasters based on the historical ranch status preprocessing data. If the conditions are met, the disaster judgment module determines that the marine ranch is in a state of meteorological disaster, hydrological disaster, or geological disaster.

[0049] Meteorological disasters, hydrological disasters, and geological disasters all have specific early warning conditions, namely the early warning interval standards corresponding to national standards. For example, storm surges, which belong to hydrological disasters, are judged to be in the event of a storm surge disaster when the above-warning tide level, wind speed, and one-third of the tide height H1 / 3 in the historical ranch status preprocessing data all exceed the early warning conditions.

[0050] In step two, the action space module includes several preset post-disaster actions. Each preset post-disaster action corresponds to a meta-action taken when a parameter value exceeds the warning value. The parameter value exceeding the warning value is one of the parameter values ​​in the historical ranch status data of the marine ranch, that is, one of the parameter values ​​contained in the marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data, such as wind speed and tide level. The preset post-disaster actions include the start and stop time, start and stop duration, movement direction, and movement speed of the equipment measuring the parameter value exceeding the warning value.

[0051] Preset post-disaster actions must be operations supported by IoT devices or data acquisition devices. Preset post-disaster actions also need to account for the lag in device operation, which reduces the overall efficiency of the actions. When multiple parameters are abnormal, it is also determined to be a certain type of disaster. For each decision step, only one decision action is selected from the action space. This action has a certain probability p that it is random (exploratory nature), and the remaining probability 1-p is to select the action with the highest current reward. A decision round consists of an action sequence composed of all actions from one or more decision steps, which can also be called a decision scheme.

[0052] The interactive environment module outputs feedback results generated by the virtual ranch sea area, specifically the ranch status data after the virtual ranch sea area takes preset post-disaster actions.

[0053] Step 3: Obtain real-time ranch status data of the marine ranch. Input the real-time ranch status data into the data processing module for data preprocessing to obtain real-time ranch preprocessed status data, and input it into the decision module. The decision module outputs preliminary decision data.

[0054] In step three, the decision module is a Deep Q-Network (DQN), specifically employing a dual-memory model LSTM. The LSTM model comprises a short-term memory network and a long-term memory network connected sequentially. The short-term memory network consists of two components: a Deep Q-Network for learning the current task and an experience replay network containing only data from the current task. The long-term memory network also comprises two components: a Deep Q-Network containing knowledge learned from all tasks since the beginning, and a generative adversarial network (GAN) for generating representations of these reinforcement learning task experiences. The decision module is constructed based on the DQN (Deep Q-learning Network) algorithm of deep learning, trained offline using an off-policy strategy, with value function approximation to the neural network construction, and employing a target network and experience replay methods for network training.

[0055] Step 4: Input the preliminary decision data into the interactive environment module. The interactive environment module outputs the predicted state value and state change of the virtual ranch sea area. Input the historical ranch state preprocessing data, the feedback results generated by the virtual ranch sea area, the predicted state value and state change into the disaster judgment module. The disaster judgment module judges whether the disaster of the marine ranch has ended and outputs the judgment result.

[0056] In practice, the disaster assessment module combines historical ranch status preprocessing data, feedback results from virtual ranch sea areas, predicted status values ​​and status changes with early warning conditions and thresholds to summarize a parameter-induced disaster correlation formula. This formula is used to determine whether the current ranch sea area and virtual sea area environment are in a risk disaster state, i.e., whether the disaster has ended. Specifically, it can determine whether the predicted status value is still within the risk range.

[0057] In step four, the preliminary decision data specifically consists of an action sequence composed of one or more preset post-disaster actions in the action space module. The preliminary decision data is input into the interactive environment module. After the interactive environment module takes the action sequence, it outputs the predicted state value and state change of the virtual ranch sea area. The predicted state value of the virtual ranch sea area is specifically the ranch state data after the virtual ranch sea area takes the action sequence. The state change of the virtual ranch sea area is the change in the ranch state data before and after taking the action sequence.

[0058] In actual processing, the interactive environment module can only receive sea area status parameters as input. Therefore, it is necessary to convert the action sequence in the preliminary decision data into an increase or decrease sequence of parameter values ​​exceeding the warning value in the pasture sea area status data of the previous decision stage before inputting it into the interactive environment module. At the same time, it is necessary to consider the differences in the parameter change rate corresponding to the action in different disaster scenarios.

[0059] When a marine ranch experiences a meteorological, hydrological, or geological disaster, and several parameter values ​​in the real-time ranch status data exceed the warning value (i.e., several parameter values ​​in the (micro)delayed data exceed the warning value), then pre-set post-disaster actions need to be taken to regulate the parameters exceeding the warning value, thus forming preliminary decision data. For example, in a storm surge disaster scenario, the parameters exceeding the warning value are wind speed, tide level, wave height, and current velocity. At this time, the pre-set post-disaster actions include the movement direction and speed of equipment for measuring wind speed, tide level, wave height, and current velocity. These actions together constitute preliminary decision data.

[0060] In step four, the disaster judgment module determines whether the disaster in the marine ranch has ended and outputs the judgment result. When all the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value do not exceed the warning value, the disaster in the marine ranch is judged to have ended. When one or more of the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value still exceed the warning value, the disaster in the marine ranch is judged to have not ended.

[0061] Step 5: Input the judgment result output by the disaster judgment module, the predicted state value of the virtual ranch sea area, and the state change into the reward update module. The reward update module calculates the reward value for this time.

[0062] In step five, the judgment results output by the disaster judgment module are input into the reward update module. The reward update module calculates the reward value for that time. Each judgment result output by the disaster judgment module represents one decision step time. When the disaster judgment module determines that the disaster in the marine ranch has not ended, it provides a negative feedback value based on the current decision step time. When the disaster judgment module determines that the disaster in the marine ranch has ended, it provides a positive feedback value based on the disaster type. When the total decision step time consumed during training exceeds the response time, the training is considered to have ended, and the decision model continues to be trained on the disaster data until the model can eliminate / avoid / reduce the risk of disaster within the response time.

[0063] The change in the state of the marine ranch is specifically the change between the real-time ranch preprocessed state data and the real-time ranch preprocessed state data after the interactive environment module takes action sequences; the environmental prediction error of the marine ranch is specifically the error between the predicted state value of the virtual ranch sea area and the real-time ranch preprocessed state data after the interactive environment module takes action sequences.

[0064] Step Six: Correct the judgment results output by the disaster judgment module and the predicted state value of the virtual ranch sea area based on the real-time ranch preprocessing state data; process the corrected judgment results and predicted state values, preliminary decision data, marine ranch state changes and environmental prediction error input parameter optimization module, and then input the processed output into the decision module for update and optimization.

[0065] During the correction process, the predicted state value of the virtual ranch sea area is corrected to the real-time ranch preprocessed state data. At the same time, the real-time state of the marine ranch is determined, and the judgment result output by the disaster judgment module is corrected to the real-time state of the marine ranch.

[0066] Step 7: Repeat steps 1 to 6 to train the disaster judgment module and decision-making module repeatedly until the reward value calculated by the reward update module converges to the maximum value. Stop training the disaster judgment module and decision-making module to obtain the trained disaster judgment module and decision-making module. Based on parameter optimization, it is also necessary to reduce the number of training rounds required.

[0067] Step 8: Real-time acquisition of marine ranch monitoring status data and input into the data processing module for data preprocessing to obtain ranch preprocessed monitoring status data. Input the ranch preprocessed monitoring status data into the trained disaster judgment module. When the disaster judgment module determines that a disaster has occurred in the marine ranch, input the ranch preprocessed monitoring status data into the trained decision module, process it, and output monitoring decision data. Make decisions for the marine ranch that has been affected by the disaster based on the monitoring decision data.

[0068] Specific embodiments of the present invention are as follows:

[0069] In practical applications, the relevant parameters in the algorithm model can be adjusted and optimized based on the real-time data transmitted back from the near-shore equipment. The specific steps include: acquiring multiple sets of real-time pasture status data; inputting each set of real-time pasture status data into the pre-trained algorithm model; combining the changes in real-time status data after decision-making to obtain the actual convergence speed; and further updating the relevant parameters of the algorithm model.

[0070] Because the marine environment itself is complex and uncertain, and marine environmental parameters can be used without feature extraction; the data returned by the monitoring equipment is time-series, which meets the characteristics of sequential decision-making in reinforcement learning; at the same time, the information obtained by the decision-making agent is completely consistent with that of the real decision-maker, and no supervision is required, which means that the final decision made by the agent may be better than that of the human decision-maker.

[0071] like Figure 2As shown, based on the characteristics of reinforcement learning and combined with the potential disaster types in marine ranches (i.e., meteorological disasters, hydrological disasters, and geological disasters), the real-time or micro-latency state, actions, decay coefficient, initial and final states, rewards, and state transition probability matrix of the underwater agent are fully customized and modeled as a quadruple.<S,A,R,T> The Markov decision process is represented by a quaternion.<S,A,R,T> The meanings of each are as follows: S—State, the current state of the environment; A—Actor (or Agent), the intelligent agent; R—Reward, the reward after the decision; T—Trajectory, a decision-making process (trajectory).

[0072] To construct a ranch disaster decision-making model, the marine environment must first be considered as the interaction environment for the agent. When the environment is partially observable rather than fully known, the observed environment is used as the input state. Next, all actions that real decision-makers can take are statistically analyzed, such as starting and stopping relevant equipment or other operations, to form a set of actions. After designing and implementing the agent's policy function and action value function, it is trained. The end of a decision-making episode is marked by the disaster being averted (i.e., all indicators returning to normal values) or exceeding the decision time limit. If the agent's actions cause abnormal parameters to revert to normal values, the reward is positive; if the disaster worsens, the reward is negative. The training process is repeated until the agent exhibits relatively stable performance.

[0073] The above process can be summarized as follows:

[0074] 1. Agent: The main body of the model;

[0075] 2. Environment: Virtual marine environment;

[0076] 3. Action: A set of actions, such as starting or stopping related equipment;

[0077] 4. Trajectory (Episode): A single sampling (i.e., a single decision).

[0078] There are two types of end markers:

[0079] 1) Disaster relief (all indicators return to normal values);

[0080] 2) Exceeding the decision-making time limit;

[0081] 5. Reward: The value gained from the action.

[0082] Evaluate the actions taken by the agent:

[0083] 1) Positive value: Outlier parameters regress to normal values;

[0084] 2) 0 or negative value: no change or abnormal exacerbation;

[0085] S1: State

[0086] S101: Marine State Transition Matrix

[0087] Let h be the history of the state of the ranch's sea area. t ={s1, s2, s3, ..., s t}(h t Including all previous states of the ranch area), s1, s2, s3, ..., s t These represent the states of the pasture sea area at times 1, 2, 3...t before the current time.

[0088] If a marine environment state transition conforms to Markov law, it means that the next state of a pasture marine area depends only on its current state and is independent of its previous states, i.e., it satisfies the following condition:

[0089] p(s t+1 |s t )=p(s t+1 |h t )

[0090] P(s t+1 |s t ,a t )=P(s t+1 |h t ,a t )

[0091] Where p() represents the state transition probability of the pasture sea area; s t+1 This represents the state of the pasture sea area at time t+1 before the current time; h t Represents all historical states of the pasture sea area before time t; a t This indicates the action selected at time t.

[0092] However, in most cases, due to limitations in equipment or sea conditions, certain parameters in the marine environment are unobservable. Nevertheless, these observable parameters can still be transformed to satisfy the MDP process. The state transition matrix P of the marine area is used to describe the state transition probability o(s). t+1 =s′|s t =s):

[0093]

[0094] Where s′ represents the state of the pasture sea area at the next moment; s N This represents the state of the pasture sea area at the current moment N moments ago.

[0095] In this embodiment, when the sea area is small, the number of data acquisition devices is limited, and the amount of data obtained is also small after averaging and sampling, the historical state data of the pasture sea area can be used as the current state in the Bellman equation. The iterative relationship between the current state and the future state is transformed into a value function relationship. The value function of each state is calculated by solving a system of equations simultaneously for all the value functions of all states. This method can bypass step S103 of establishing and applying the sea area model.

[0096]

[0097] Where V() represents the state value function; R() represents the reward function; and γ represents the discount factor.

[0098] S102: State Space

[0099] The application scenario of this embodiment can be any marine ranching risk disaster scenario, such as typhoon waves, storm surges, etc. For example, if the application scenario is a storm surge scenario, then the current state data can be the current velocity data (including layer number, depth, raw velocity data, x-direction velocity, y-direction velocity, z-direction velocity, composite velocity, composite velocity direction, etc.) and ecological simulation forecast data (including time, longitude, latitude, depth, water level, salinity, water temperature, eastward velocity, and northward velocity), which can be obtained through sensor feedback or ecological numerical model simulation. This embodiment does not limit the application scenario or the method of data acquisition; specific implementation can be constructed according to the needs of the scenario.

[0100] S103: Interactive Environment

[0101] The equipment deployed in marine ranches can be mainly divided into three categories: aquaculture equipment, monitoring equipment, and IoT equipment. Among them, aquaculture equipment accounts for the largest share, exceeding 90% in most ranches.

[0102] In boundary delineation, equipment density has a significant impact on the model's prediction and assessment of current water area parameter trends. Delineation can be done horizontally and vertically. Horizontally, three types of boundary conditions can be applied, from highest to lowest density: enclosed lake conditions, semi-enclosed sea conditions, and open sea conditions. Vertically, reasonable stratification is required based on the type of aquaculture equipment and the species of aquaculture. Single-layer aquaculture equipment does not require vertical boundary delineation, while multi-layer aquaculture requires three layers: water-air contact surface, shallow water, and deep sea. The interactive environment module is based on the parameter prediction sections for semi-enclosed sea areas and open oceans in the Princeton Ocean Model (POM).

[0103] In this embodiment, the process of building the interactive environment is as follows: Figure 3 As shown. Based on the ranch planning scheme (including aquaculture type, aquaculture area, and suitable environmental parameter range), driving factor parameters are set, and historical ranch data is sampled together and input into a neural network based on an ANN (Artificial Neural Network) model. The output is the land use probability of this sea area, which is part of the total probability. The other part of the total probability consists of the product of the transition matrix, neighborhood, and adaptive inertia. A random seed is defined, a roulette wheel selection is performed, and the simulation results are output. Through value judgment based on a Markov prediction chain, the result is output when the simulation result meets the requirements; otherwise, the adaptive inertia ratio is adjusted until the result meets the requirements.

[0104] S2: Decision Maker (Agent / Actor)

[0105] The decision-making module is a Deep Q-Network (DQN), specifically employing a dual-memory LSTM model. The LSTM model consists of a short-term memory network and a long-term memory network connected sequentially. The short-term memory network comprises two components: a Deep Q-Network for learning the current task and an experience replay network containing only data from the current task. The long-term memory network also comprises two components: a Deep Q-Network containing knowledge learned from all tasks since the beginning, and a generative adversarial network (GAN) for generating representations of these reinforcement learning task experiences. The decision-making module is constructed based on the Q-learning algorithm of deep learning, using an off-policy offline training method. The value function approximates the neural network construction, and the network training employs a target network and experience replay techniques.

[0106] S3: Reward

[0107] S301: Reward Function

[0108] The most crucial factor affecting agent behavior during the decision-making process is the reward function R. The reward function R is an expectation; it represents the reward obtained when a certain state is reached. The reward for future states is typically multiplied by a discount factor γ.

[0109] G t =R t+1 +γR t+2 +γ 2 R t+3 +γ 3 R t+4 ++γ T-t-1 R T

[0110] Among them, G t R represents the total reward value at time t; t+1 R t+2 R t+3 R t+4 ...R T Let represent the reward functions at times t+1, t+2, t+3…T respectively.

[0111] Since simulated environments cannot perfectly replicate real-world environments, there is uncertainty in the evaluation of future states during the decision-making process. Furthermore, to ensure the agent receives rewards as quickly as possible, rather than at a later point, a discount factor diminishes the rewards from future states, indicating to the agent that the current reward is more important.

[0112] In this embodiment, the discount factor γ is set to 0.95. The actual value can be adjusted according to the needs of specific disaster application scenarios. In particular, when γ is 1, it means that there is no discount on future rewards, and the rewards obtained in the future are equally important as the rewards obtained now; when γ is 0, it means that only the immediate rewards are considered, and future rewards are completely ignored.

[0113]

[0114] After determining the state transition matrix of the pasture sea area based on the Markov chain, sampling the chain yields a series of trajectories. The reward process can be understood as the superposition of the Markov chain and the reward function R:

[0115]

[0116] Among them, V t () represents the state value function at time t; 's' indicates expectation; 's' indicates the current state of the ranch area.

[0117] Specifically, the state value function V is calculated. πThe calculation of (s) requires the use of the Bellman equation:

[0118]

[0119] Where S represents the set of states of the ranch sea area at the next moment.

[0120] There are two different approaches to evaluating the state value function: one is the Monte Carlo sampling (MC-based) method, and the other is the temporal difference (TD-based) method. In the Monte Carlo sampling approach, after obtaining an MRP (Master Reward Program), multiple trajectories can be sampled starting from a certain state, and the discounted reward g for each trajectory can be calculated to obtain the total reward G. t The value of a state can be obtained by approximating its value by dividing it by the number of trajectories.

[0121] S302: Special Bonus Value

[0122] In this embodiment, the optimization objective is to eliminate the current risk and disaster state or escape the current risk and disaster environment in the shortest possible time. Therefore, the reward function can be set to give a fixed negative value as a penalty every decision time step, regardless of whether a decision is made. It can also be set according to the actual objective. For example, the reward function can be composed of equivalent weights such as disaster type, disaster factor deviation value, and decision feedback.

[0123] In this embodiment, taking the storm surge disaster scenario as an example, the special reward value is set as shown in Table 1 below. According to the level range of the disaster, the reward value is updated accordingly for each decision step.

[0124] Table 1. Reward value settings for storm surge disaster scenarios in the examples.

[0125]

[0126]

[0127] S4: Single Decision Round (Trajectory / Episode) and Repeated Training

[0128] S401: General Training Procedures

[0129] like Figure 4As shown, historical ranch status data is monitored and captured using Python scripts and input into a data processing module for data preprocessing to obtain historical ranch preprocessed status data. Specifically, the historical ranch status data is input into the data processing module for missing value imputation, random sampling, and serialization processing. In practice, missing values ​​are imputed for data with gaps. Furthermore, each set of ranch status data is processed based on depth. For small datasets, vertical averaging is used, while for large datasets, a VAE model is used for data compression. The processed outputs are combined to form the historical ranch preprocessed status data and stored in a MongoDB database.

[0130] Historical pasture preprocessing status data are input into the interactive environment model of the marine ecological simulation assessment model in time series for dynamic evaluation. The decision time step is limited to 1 / 1000 of the data update time (the data update time interval is 5 minutes during disasters and 1 hour in other cases). The initial reward is set to 0. If the disaster is still ongoing after each decision time step, the reward value is reduced by 1. Abnormal values ​​in historical state data are adjusted. The action space (increase / decrease values) is continuous (the value range needs to be calculated according to the parameter change curve, i.e., according to the change law of marine hydrodynamic parameters). This mainly includes using deployed underwater equipment to regulate the monitorable and adjustable marine parameters. Actions are input into the interactive environment model to update the prediction data. The prediction data is then input into the disaster judgment module to calculate whether the current risk disaster has been resolved. If there is delayed real-time data (due to hardware reasons), it can be used to correct the prediction data output by the model and update the disaster state. When the risk disaster is resolved, the reward value is increased by 500 (which can be used for parameter tuning). Training is repeated until the reward value converges to its maximum value. Under real-world conditions, real-time data is sampled, serialized, and input into the trained algorithm model. Simultaneously, decision feedback from the actual marine environment is collected to adjust and update parameters, and the correctness of the algorithm model is verified. Experiments are repeated to optimize the strategy.

[0131] During training, historical state data and decision strategies from the ranch can be used as input to the BNN (Bayesian Neural Network), while state changes and reward values ​​are used as outputs. The BNN is trained iteratively, with state changes representing the difference between the next and previous states. This design directly impacts the amount of data required for Bayesian training and its effectiveness. If the output is the next state data, it's equivalent to Bayesian training learning a complete mapping from one state to another, requiring far more data than training on state changes. By training a Bayesian neural network with a small amount of historical data, a virtual environment can be constructed for the entire ranch's marine environment, providing more training data for the reinforcement learning model.

[0132] To address the potential convergence issue of deep Q-networks in certain disaster scenarios, and considering the inherent latency of real-time data transmitted from actual sea areas, this embodiment employs a fixed Q-targets method with delayed parameter updates. This results in the DQN having two networks: a Predict Q Network and a Target Q Network. The Predict Q Network predicts the Q-values ​​for each action in the current state, while the Target Q Network predicts the Q-values ​​for each action in the next state, or the nth state after that. The parameters in the Predict Q Network are updated in real-time, and the Target Q Network determines whether to update based on the parameter updates from the Predict Q Network, thereby eliminating back interference caused by abnormal environmental parameters or complex causes of disasters in disaster scenarios.

[0133] S402-S403: Special Scenario Training and Real-Time Decision-Making Scenario Training

[0134] Since this embodiment uses an off-policy offline training method, and combines it with the experience replay method, it caches the state, action, reward, and next state tuples for each step, and performs batch training multiple times after each round, which improves the training speed and stability of DQN. Specifically, it maintains a cache array of a specified size, and randomly replaces the existing N tuples in the cache pool with the newly generated N state, action, reward, and next state tuples in each round, and then performs several training cycles after the round ends.

[0135] This embodiment employs an off-policy offline training method for a marine ranching disaster decision-making algorithm model based on reinforcement learning. This means the algorithm model, already trained using the aforementioned real-time method, is ready for real-time use. In practical applications, relatively accurate decision data can be obtained directly by inputting the current state data. When the decision scenario is a storm surge disaster, the reinforcement learning model, already trained offline using the method described in the previous embodiment, only requires inputting the current state data to obtain decision data that meets the optimization objective. The trained algorithm model's decision objective is to alleviate the current risk disaster state or escape the current risk disaster environment in the shortest possible time; the decision data then guides the decision-making process to achieve this objective.

[0136] During each training round, the generative adversarial network (GAN) is retrained to generate samples that simultaneously represent previous GAN experiences and current task experience. Similar to short-lived deep Q-networks, its training method and loss function simply follow standard GAN practices. The construction of decision models for practical applications requires research based on related prefix topics, such as identifying disaster-causing factors and the specific operation of monitoring equipment, to construct training sets and action sets. In actual training, combining the marine ecological numerical model constructed from the prefix topics as a virtual marine environment significantly improves both the training speed and decision-making performance of the model.

[0137] Reinforcement learning, as a paradigm and methodology of machine learning, is based on the idea of ​​influencing the state of the environment by applying actions and learning the optimal strategy to achieve a goal by perceiving the environment's response to those actions. The task of reinforcement learning is to learn how to map the current environmental state to actions to maximize the reward signal. The agent, environment, policy, reward signal, value function, and environment model constitute the basic elements of reinforcement learning. This invention provides a reinforcement learning-based disaster decision-making method for ranches, enabling the agent to autonomously determine how to adjust its actions to solve the current problem based on historical data and real-time data with minimal latency from existing ranch and marine monitoring indicators, ultimately achieving self-learning.

[0138] This invention combines an interactive environment model constructed from complex historical data of marine ranches. Through correlation analysis of disaster-causing factors and calculation of action feedback using a disaster classification system, it determines the action space and corresponding implementation strategies, enabling real-time decision-making and strategy evaluation for risks and disasters involved in marine ranches. This provides a foundation for subsequent scientific research, ranch layout optimization, and the construction of ranch disaster response systems, promoting the sustainable development of marine ranches.

Claims

1. A disaster decision-making method for marine ranching based on reinforcement learning, characterized in that: The method includes the following steps: Step 1: Obtain the historical ranch status data of the marine ranch before the current time. Input the historical ranch status data into the data processing module for data preprocessing to obtain the historical ranch preprocessed status data. Input the historical ranch status preprocessed data into the interactive environment module to construct the virtual ranch sea area of ​​the marine ranch in the interactive environment module. Step 2: Input the preprocessed historical ranch status data into the disaster judgment module. The disaster judgment module determines whether a disaster has occurred in the marine ranch. When a disaster occurs in the marine ranch, a preset post-disaster action is input into the interactive environment module through the action space module to take the preset post-disaster action on the virtual ranch sea area. The interactive environment module outputs the feedback results generated by the virtual ranch sea area. Step 3: Obtain real-time ranch status data of the marine ranch, input the real-time ranch status data into the data processing module for data preprocessing to obtain real-time ranch preprocessed status data, and input it into the decision module. The decision module outputs preliminary decision data. Step 4: Input the preliminary decision data into the interactive environment module. The interactive environment module outputs the predicted state value and state change of the virtual ranch sea area. Input the historical ranch state preprocessing data, the feedback results generated by the virtual ranch sea area, the predicted state value and state change into the disaster judgment module. The disaster judgment module determines whether the disaster of the marine ranch has ended and outputs the judgment result. Step 5: Input the judgment result output by the disaster judgment module, the predicted state value and state change of the virtual ranch sea area into the reward update module, and the reward update module calculates the reward value for this time; Step Six: Correct the judgment results output by the disaster judgment module and the predicted state value of the virtual ranch sea area based on the real-time ranch preprocessing state data; process the corrected judgment results and predicted state values, preliminary decision data, marine ranch state changes and environmental prediction error input parameter optimization module, and then input the processed output into the decision module for update and optimization; Step 7: Repeat steps 1 to 6 to train the disaster judgment module and decision-making module repeatedly until the reward value calculated by the reward update module converges to the maximum value. Stop training the disaster judgment module and decision-making module and obtain the trained disaster judgment module and decision-making module. Step 8: Real-time acquisition of marine ranch monitoring status data and input into the data processing module for data preprocessing to obtain ranch preprocessed monitoring status data. Input the ranch preprocessed monitoring status data into the trained disaster judgment module. When the disaster judgment module determines that a disaster has occurred in the marine ranch, input the ranch preprocessed monitoring status data into the trained decision module, process it, and output monitoring decision data. Make decisions on the marine ranch that has been affected by the disaster based on the monitoring decision data. In step one, the interactive environment module constructs a virtual ranch sea area based on the preprocessed data of the historical ranch status, the deployment layout structure of each device in the marine ranch cluster, and the two-dimensional shallow water equation and embedded second-order moment turbulent closure sub-model of the sea area where the marine ranch is located. In step three, the decision module is a deep Q-network (DQN), which specifically adopts a dual-memory model (LSTM). The dual-memory model (LSTM) includes a short-term memory network and a long-term memory network connected in sequence. In step four, the preliminary decision data specifically consists of an action sequence composed of one or more preset post-disaster actions in the action space module. The preliminary decision data is input into the interactive environment module. After the interactive environment module takes the action sequence, it outputs the predicted state value and state change of the virtual ranch sea area. The predicted state value of the virtual ranch sea area is specifically the ranch state data after the virtual ranch sea area takes the action sequence. The state change of the virtual ranch sea area is the change in the ranch state data before and after taking the action sequence.

2. The marine ranching disaster decision-making method based on reinforcement learning according to claim 1, characterized in that: The historical and real-time ranch status data of the marine ranch mentioned above include multi-parameter sensor data, turbidity sensor data, flow velocity data, and ecological simulation and forecast data of the marine area; Historical pasture status data is input into the data processing module for data preprocessing to obtain historical pasture preprocessed status data. Specifically, the marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data from the historical pasture status data are input into the data processing module for missing value supplementation, random sampling, and serialization processing in sequence. The processed outputs are used to construct the historical pasture preprocessed status data.

3. The marine ranching disaster decision-making method based on reinforcement learning according to claim 1, characterized in that: In step two, the disasters that occur in the marine ranch specifically include meteorological disasters, hydrological disasters, and geological disasters. The historical ranch status preprocessing data is input into the disaster judgment module, which determines whether the marine ranch has experienced a disaster. Specifically, the disaster judgment module determines whether the marine ranch meets the early warning conditions for meteorological disasters, hydrological disasters, or geological disasters based on the historical ranch status preprocessing data. If the conditions are met, the disaster judgment module determines that the marine ranch is in a state of meteorological disaster, hydrological disaster, or geological disaster.

4. The marine ranching disaster decision-making method based on reinforcement learning according to claim 3, characterized in that: In step two, the action space module includes several preset post-disaster actions. Each preset post-disaster action corresponds to a meta-action taken when a parameter value exceeds the warning value. The parameter value exceeding the warning value is one of the parameter values ​​in the historical ranch status data of the marine ranch, namely, one of the parameter values ​​included in the marine multi-parameter sensor data, turbidity sensor data, current velocity data, and ecological simulation forecast data. The preset post-disaster actions include the start-up and stop times, start-up and stop durations, movement direction, and movement speed of the equipment measuring the parameter value exceeding the warning value. The feedback results generated by the interactive environment module are specifically the ranch status data after the virtual ranch sea area takes preset post-disaster actions.

5. A marine ranching disaster decision-making method based on reinforcement learning according to claim 3, characterized in that: In step four, the disaster judgment module determines whether the disaster in the marine ranch has ended and outputs the judgment result. When all the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value do not exceed the warning value, the disaster in the marine ranch is judged to have ended. When one or more of the parameter values ​​in the predicted state value of the virtual ranch sea area that exceed the warning value still exceed the warning value, the disaster in the marine ranch is judged to have not ended.

6. The marine ranching disaster decision-making method based on reinforcement learning according to claim 1, characterized in that: In step five, the judgment result output by the disaster judgment module is input into the reward update module. The reward update module calculates the reward value for the current time. Each judgment result output by the disaster judgment module is equivalent to consuming one decision step time. When the disaster judgment module determines that the disaster of the marine ranch has not ended, it gives a negative feedback value based on the current decision step time. When the disaster judgment module determines that the disaster of the marine ranch has ended, it gives a positive feedback value based on the disaster type of the marine ranch.

7. The marine ranching disaster decision-making method based on reinforcement learning according to claim 1, characterized in that: The state change of the marine ranch is specifically the change between the real-time ranch preprocessed state data and the real-time ranch preprocessed state data after the interactive environment module takes the action sequence; the environmental prediction error of the marine ranch is specifically the error between the predicted state value of the virtual ranch sea area and the real-time ranch preprocessed state data after the interactive environment module takes the action sequence.

Images