Buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHERN MARINE SCI & ENG GUANGDONG LAB (ZHUHAI)
- Filing Date
- 2026-05-19
- Publication Date
- 2026-07-21
Smart Images
Figure CN122431091A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of photovoltaic panel control technology based on reinforcement learning, and particularly relates to a buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning. Background Technology
[0002] With the continuous advancement of marine renewable energy development, offshore distributed photovoltaic (PV) systems, as an emerging form of energy utilization, are gradually attracting attention. Compared to terrestrial PV, marine buoy-based PV systems offer advantages such as flexible deployment, minimal land resource conflicts, and integration with marine monitoring equipment, demonstrating broad application prospects in marine observation, communication relay, and offshore power supply. However, the complex and ever-changing marine environment places higher demands on the structural safety and power generation efficiency of PV systems. Especially under the coupled effects of strong winds and waves, complex current fields, and radiation variations, traditional fixed structures or simple control strategies struggle to achieve an effective balance between safety and energy efficiency. Most existing buoy-based PV systems employ fixed tilt angles or simple rule-based control methods, such as opening and closing control based on wind speed thresholds or wave height levels. While these methods are simple to implement, they have significant limitations. On one hand, the marine environment exhibits high randomness and nonlinearity, making it difficult for a single threshold or empirical rule to accurately reflect the actual stress state, easily leading to control decision lags or failures. On the other hand, this coarse-grained control cannot fully utilize favorable radiation conditions in the environment, resulting in low solar panel light absorption efficiency and thus limiting overall power generation performance. Furthermore, frequent or unreasonable folding and unfolding movements can increase fatigue damage to the mechanical structure and shorten the equipment's service life. From a control theory perspective, the folding and unfolding process of the buoy photovoltaic panel is essentially a typical multivariable coupled, highly nonlinear, dynamic control problem with uncertain disturbances. The system state is not only affected by external sea conditions (waves, wind field, current field) but also closely related to internal structural parameters (angle, stress distribution). Simultaneously, the control objectives involve multiple constraints, including maximizing power generation efficiency, ensuring structural safety, and minimizing energy consumption. Traditional control methods (PID control or model-based optimization control) often rely on precise mathematical models in such complex systems. However, the high uncertainty of the marine environment makes it difficult to accurately establish models, thus limiting further improvements in control performance.
[0003] In recent years, deep reinforcement learning, as an intelligent decision-making method combining deep neural networks and reinforcement learning mechanisms, has demonstrated significant advantages in the control of complex dynamic systems. By continuously learning optimal strategies through interaction with the environment, it can handle high-dimensional state spaces and nonlinear decision-making problems without the need for precise modeling, making it particularly suitable for control scenarios with stochastic and time-varying characteristics. However, existing research has largely focused on areas such as robot control, autonomous driving, and smart grids, with limited application in marine buoy photovoltaic systems. Furthermore, directly applying standard reinforcement learning methods to engineering systems faces challenges such as the difficulty in explicitly embedding safety constraints and discrepancies between training data and the real environment, affecting its practical implementation. In addition, marine equipment is typically characterized by remote deployment and high maintenance costs; improper control strategies leading to structural damage can result in substantial economic losses. Therefore, safety and reliability must be fully considered in control strategy design to avoid irreversible damage to equipment caused by improper behavior during strategy exploration. This requires control methods to not only possess adaptive optimization capabilities but also introduce safety constraint mechanisms and continuously correct model deviations during actual operation to achieve long-term stable operation.
[0004] Against this backdrop, there is an urgent need for an intelligent control method that can adapt to complex marine environments, balance power generation efficiency and structural safety, and possess self-learning and online optimization capabilities. Summary of the Invention
[0005] To address the above problems, this invention provides a buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning, comprising the following processes: S1, the original state vector is constructed by real-time acquisition of marine environment and photovoltaic panel structure status through multi-source sensor array, and a unified state vector is constructed after preprocessing and trigonometric function mapping of direction angle variable; S2 uses a unified state vector as input to perform multi-physics coupled dynamics modeling and formal modeling of Markov decision process. It constructs a multi-objective evaluation mechanism through power generation, structural stress and control energy consumption to form an environmental interaction interface to obtain the theoretical optimal state quantity. Then, it introduces a state transition proxy model to obtain the predicted state. After weighted fusion, it outputs the optimal state prediction result to construct a high-fidelity simulation environment. S3, based on the optimal state prediction results, outputs control actions through the policy network and constructs a state value update value network to perform collaborative optimization among power generation revenue, energy consumption and structural safety constraints, and finally outputs the optimal expanded control action and corresponding state value. S4 deploys the strategy network in the buoy's local control unit, receives the unified state vector in real time, and outputs the optimal unfolded control action by the strategy network.
[0006] Preferably, the original state vector includes: On the environmental side, wave radar acquires effective wave height. With wave cycle Flow velocity and direction meter acquire flow velocity With the angle of flow Wind speed and direction are obtained from the anemometer. with wind angle And radiometers to count solar radiation intensity ; On the structural side, the folding angle of the photovoltaic panel is obtained by using an angle encoder and strain gauges, respectively. angular velocity ; The system uses a fixed sampling period Perform discrete sampling, at the... The original state vector is formed at each moment. .
[0007] Preferably, the preprocessing includes time synchronization, linear interpolation and denoising, and normalization; for variable flow direction angles related to the direction angle... and wind direction angle It is processed using triangular mapping.
[0008] Preferably, the multiphysics coupling dynamics modeling in S2 specifically includes: First, a marine environmental excitation model is constructed to obtain the wave force expression: ; in The density of seawater, It is the acceleration due to gravity; For the effective wave height, For wave cycles, This is the current time variable.
[0009] Meanwhile, wind load With fluid load They are modeled as follows: ; ; in, air density, and The drag coefficient, The area under stress; For wind speed, This refers to the ocean current velocity.
[0010] Three forces through lever arm Converting to torque and then superimposing the directions yields the environmental torque: ; lever arm This represents the equivalent distance between the point of force application on the photovoltaic panel and its axis of rotation. On the structural side, the photovoltaic panel is simplified as a rotating rigid body system, and its dynamic equation is: ; in For rotational inertia, The damping coefficient is... For equivalent stiffness, To control the input torque; Let be the angular acceleration of the photovoltaic panel's folding angle. The angular velocity is the adjustment amount of the photovoltaic panel's folding angle through reinforcement learning feedback. Proceed to the next status update.
[0011] Preferably, the formal modeling of the Markov decision process specifically includes: Defined as a quadruple ;in: state space Constructed sequence sample data composition; Action space Adjustment amount for the folding angle of the photovoltaic panel ; State transition function Based on the constructed dynamic equations, for Solve the equation and obtain the solution. ,in This represents the solution obtained from the dynamic equations for calculation. Combination mapping function; and As a physical state transition function, and This is the theoretical state quantity of the photovoltaic panel's folding angle; reward function To guide control strategies to achieve a balance between power generation efficiency, energy consumption, and structural safety: ; in, Photovoltaic power generation capacity, For structural stress response, To control energy consumption, the adjustment of the photovoltaic panel's folding angle is used. With energy consumption coefficient The product is obtained; The safety penalty term is then defined as a piecewise function: when hour, ;when hour, ; , , The importance weights are assigned to power generation revenue, energy consumption penalties, and security constraints.
[0012] Preferably, a feedforward neural network is constructed as a state transition proxy model, and the model input includes sequential sample data. Adjustment amount of the folding angle compared to the previous moment Afterwards, the network passes through hidden layers, including three fully connected layers. Each fully connected layer is followed by a ReLU activation function to initially extract non-linear features of the environment. Then, it passes through an output layer containing a linear activation function to output the predicted state. .
[0013] Preferably, in step S3, a high-fidelity simulation environment is constructed. As input to the training environment, the environment can return the optimal state prediction result at each step of the interaction. With instant rewards ; at each time step Execute the following interaction flow: Input the current state vector Policy network output actions Environmental execution Return the optimal state prediction result With rewards Storage state transition samples The trajectory cache pool is used to construct an Actor-Critic framework based on near-end policy optimization of PPO, which learns the optimal control policy from interaction data.
[0014] Preferably, the policy network consists of an input embedding layer, a feature transformation layer, a gated fusion layer, and a policy output layer: the input embedding layer receives the optimal state prediction result. Linear mapping and normalization are performed to map the original physical quantities to a unified feature space; the feature transformation layer contains two fully connected layers and introduces layer normalization to improve training stability; the gated fusion layer adopts a gated linear unit structure to adaptively weight and fuse environmental and structural features; the policy output layer adopts a dual-branch structure, with one branch outputting the action mean. The other output motion standard deviation The standard deviation is constrained to a positive value using the Softplus function; subsequently, the mean is... with standard deviation As parameters, a Gaussian probability distribution of continuous actions is constructed, and control actions are obtained by sampling from this distribution. This forms the policy function: ; The value network structure is the same as the policy network, consisting of an input layer and two fully connected hidden layers. The output layer outputs a scalar value to represent the state vector. Corresponding state value .
[0015] Preferably, it also includes an online reinforcement learning mechanism to incrementally update the policy network: extracting sample data from the experience replay pool, and optimizing the current policy network based on the established PPO policy optimization method. Small-compensation incremental updates are performed to continuously optimize the system based on real-world data. During these updates, actual operational data is used to refine the original parameters. Make corrections; at the same time, adjust the value network. Perform synchronized updates.
[0016] Preferably, it also includes a multi-layered security constraint mechanism: First, control actions Set amplitude limits to ensure that they meet the physical constraints of the actuator. Secondly, real-time monitoring of structural stress. When its When this occurs, a safety control mechanism is triggered, forcibly correcting the original control action by scaling it proportionally. Force correction, action after correction The original action input actuator is replaced, reducing the stress level on the structure and avoiding dangerous working conditions. At the same time, under extreme sea conditions, the system enters the safety protection model, directly overriding the strategy output, adjusting the photovoltaic panel to the preset safe angle, and suspending the strategy control output.
[0017] Compared with the prior art, the present invention has the following innovative features: (1) A multi-source sensing state modeling method for complex marine environments is proposed, which integrates multi-dimensional information such as waves, wind field, flow field and structural stress into a high-dimensional state space to achieve a refined characterization of the operating state of the buoy photovoltaic system. (2) A buoy photovoltaic folding control model based on Markov decision process is constructed, which couples environmental dynamics with control behavior in the model, providing a systematic decision framework for reinforcement learning algorithms. A proximal policy optimization algorithm is introduced to achieve folding angle optimization control in continuous action space, breaking through the limitations of traditional discrete or rule-based control methods; (3) Design a multi-objective reward function that integrates power generation efficiency, execution energy consumption and structural safety to achieve collaborative optimization control under multiple constraints.
[0018] The beneficial effects brought about by the innovation of this invention include: (1) Significantly improve power generation efficiency: By dynamically adjusting the angle of light received by the photovoltaic panel, the system can maintain a better energy capture state under different irradiance and sea conditions; (2) Improve structural safety: Introduce stress constraints in control decisions to effectively reduce the risk of structural damage under strong wind and wave conditions and extend the service life of equipment; (3) Enhanced environmental adaptability: Compared with fixed strategies or threshold control methods, the present invention can adapt to highly random and nonlinear changes in the marine environment. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the following description is only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the overall technical process of the present invention.
[0021] Figure 2 Formal modeling diagram of Markov decision process.
[0022] Figure 3 This is a diagram of the Actor-Critic framework based on near-end strategy optimization.
[0023] Figure 4 This is a comprehensive analysis diagram of the dynamic response of multiple physics fields in the embodiment.
[0024] Figure 5 The figure shows a simplified simulation result of the PPO strategy training in the example.
[0025] Figure 6 This is a timing dynamic response diagram in the embodiment. Detailed Implementation
[0026] By constructing a high-fidelity simulation environment, introducing multi-source sensing information, and combining deep reinforcement learning with safety constraint mechanisms, it is expected to overcome the limitations of traditional methods in nonlinearity, multi-objective optimization, and environmental uncertainty, thereby achieving intelligent and refined control of the buoy photovoltaic panel's expansion and contraction. Therefore, this invention discloses a buoy photovoltaic panel expansion and contraction control method based on deep reinforcement learning, the overall process of which is as follows: Figure 1 As shown: S1 Data Acquisition and State Space Construction. Real-time data acquisition of the marine environment and photovoltaic panel structure of the buoy is achieved using a multi-source sensor array, constructing the initial state vector. Subsequently, to address the issue of inconsistent sampling frequencies among the sensors, time synchronization and linear interpolation were performed to compensate for abnormal or missing data, resulting in a time-aligned sequence. Further denoising of the data yields a smoothed sequence. Finally, the state variables are normalized and the direction angle variables are mapped using trigonometric functions to construct a unified state vector. This yields standardized input data that can be used for subsequent reinforcement learning and dynamics modeling.
[0027] Construction of S2 high-fidelity simulation environment and decision-making model. Based on the unified state vector obtained in stage S1. As input, a Markov decision process is introduced to formally model the system, using power generation... Structural stress With control of energy consumption Construct a multi-objective evaluation mechanism and form an environment interaction interface based on physical mechanisms. Used to obtain the theoretically optimal state variables Furthermore, a state transition proxy model is introduced to obtain the predicted state. and with the theoretically optimal state quantity By performing weighted fusion, the optimal state prediction result is finally obtained. This process is constructed as a high-fidelity simulation environment. .
[0028] S3 is based on PPO-based safety constraint deep reinforcement learning strategy training. A high-fidelity simulation environment is constructed based on S2. The obtained optimal state prediction results Through policy network Output and And sample the control actions Then design state value This is used to update the value network, enabling the strategy to achieve synergistic optimization among power generation revenue, energy consumption, and structural safety constraints, ultimately outputting the optimal unfolded control action. ; Corresponding state value .
[0029] S4 closed-loop control execution and online adaptive optimization. A policy network is deployed in the buoy's local control unit, receiving state vectors in real time during operation. The optimal unfolded control action is output by the policy network. The system drives a stepper motor to perform the folding and unfolding action of the photovoltaic panel; at the same time, the system continuously collects the power generated after the action. With structural stress Wait for feedback information to construct state transition samples in a real-world environment. The experience is stored in the experience replay pool; then, an online incremental update mechanism is introduced to update the policy network. With value function Continuous adjustments are made, and action limiting and stress threshold protection mechanisms are introduced at the execution layer to ensure that the system can achieve long-term stable, safe and adaptive expansion control operation in complex marine environments.
[0030] The implementation process of the present invention will be further described below with reference to specific embodiments.
[0031] S1. Data Acquisition and State Space Construction S1-1 Multi-source sensor data acquisition and raw data construction. A multi-source heterogeneous sensor array is deployed on the buoy platform to form an integrated acquisition system for environmental perception and structural monitoring.
[0032] On the environmental side, wave radar acquires effective wave height. With wave cycle Flow velocity and direction meter acquire flow velocity With the angle of flow Wind speed and direction are obtained from the anemometer. with wind angle And radiometers to count solar radiation intensity .
[0033] On the structural side, the folding angle of the photovoltaic panel is obtained by using an angle encoder and strain gauges, respectively. angular velocity .
[0034] The system uses a fixed sampling period Perform discrete sampling, at the... The original state vector is formed at each moment. : .
[0035] S1-2 Time Synchronization and Data Alignment Processing. Due to the inconsistent sampling frequencies of different sensors, it is necessary to... All components are unified to the same time base. This is achieved through linear interpolation mapping to a unified time. For data with packet loss or abnormal delays, a sliding window mean compensation is used to obtain a time-aligned sequence. This ensures that all state variables have a consistent physical correspondence at the same time.
[0036] S1-3 Noise Reduction Filtering and Outlier Handling. This involves processing time-aligned data. For each state variable in the sequence, an exponential smoothing filter is used for filtering. .in This is the smoothing coefficient.
[0037] Further adopting a length of The moving average method enhances stationarity, resulting in a smoothed sequence after denoising. , This is the index value.
[0038] S1-4 State Normalization and State Space Construction. To eliminate the influence of dimensions, the following steps are performed: All state variables in the process are normalized to obtain ; Next, regarding the variables related to the direction angle... Triangular mapping is used for processing: Finally, a unified state vector is constructed. .
[0039] S2. Construction of High-Fidelity Simulation Environment and Decision Model S2-1 Multiphysics Coupled Dynamics Modeling. Using the unified state vector output from S1. As input, multiphysics modeling is performed on the buoy photovoltaic system.
[0040] First, a marine environmental excitation model is constructed to obtain the wave force expression: .in The density of seawater, This is the acceleration due to gravity. For the effective wave height, For wave cycles, This is the current time variable.
[0041] Meanwhile, wind load With fluid load They are modeled as follows: .in, air density, and The drag coefficient, The area under stress. For wind speed, This refers to the ocean current velocity.
[0042] Three forces through lever arm Converting to torque and then superimposing the directions yields the environmental torque. Lever arm This represents the equivalent distance between the point of force application on the photovoltaic panel and its rotation axis.
[0043] On the structural side, the photovoltaic panel is simplified as a rotating rigid body system, and its dynamic equation is: .in For rotational inertia, The damping coefficient is... For equivalent stiffness, To control the input torque (determined by the reinforcement learning action): ,in This refers to the action output for subsequent reinforcement learning, namely the adjustment amount of the photovoltaic panel's folding angle; (This refers to the torque conversion factor of the actuator). Let be the angular acceleration of the photovoltaic panel's folding angle. The angular velocity is the adjustment amount of the photovoltaic panel's folding angle through reinforcement learning feedback. Proceed to the next status update.
[0044] S2-2 Formal Modeling Based on Markov Decision Processes. After completing the construction of the dynamic model, the buoy photovoltaic expansion control problem is modeled as a Markov Decision Process (MDP) to describe the sequential decision-making process of the system in an uncertain marine environment.
[0045] Specifically, it is defined as a quadruple. .in: state space Sequence sample data constructed by S1 composition; Action space Adjustment amount for the folding angle of the photovoltaic panel ; State transition function Based on the dynamic equations constructed in S2-1, for Solve the equation and obtain the solution. ,in This represents the solution obtained from the dynamic equations for calculation. Combination mapping function; and As the initial physical state transition model, and This is the theoretical state quantity of the photovoltaic panel's folding angle.
[0046] Next, a calculation model for photovoltaic power generation and structural stress was designed. Specifically, the photovoltaic power generation capacity is: ,in, For photoelectric conversion efficiency, The theoretical state variables representing the photovoltaic panel's bending angle affect power generation efficiency. Secondly, the structural stress response model uses a linear approximation model: .in, , and These are the angle-to-stress sensitivity coefficient, the angular velocity influence coefficient, and the environmental load coupling coefficient, respectively. For comprehensive environmental load.
[0047] reward function To guide the control strategy to achieve a balance between power generation efficiency, energy consumption, and structural safety: ,in To control energy consumption, the adjustment of the photovoltaic panel's folding angle is used. With energy consumption coefficient The product is obtained; The safety penalty term is then defined as a piecewise function: when hour, ;when hour, . , , The importance weights are assigned to power generation revenue, energy consumption penalties, and security constraints.
[0048] Due to the adjustment of the photovoltaic panel's folding angle As a control variable, it has a large number of selectable values, therefore... Extensive searches were conducted to obtain a large number of theoretical state quantities for the folding angle of photovoltaic panels. The result is then used with a reward function. Evaluate these results and finally select the reward function. The theoretical state quantity of the maximum photovoltaic panel folding angle As the only determined theoretically optimal state quantity .
[0049] This step achieves a unified representation from the physical system to the decision model, and outputs an environment interface that can be used for reinforcement learning interaction. ,in This represents the Markov decision process constructed by S2-2.
[0050] S2-3 State Transition Proxy Model Construction and High-Fidelity Environmental Correction. In actual marine environments, relying solely on physical models is insufficient to fully characterize complex nonlinear coupling relationships. Therefore, a state transition Proxy model is introduced to approximate and correct the state transition process. A feedforward neural network is constructed as the state transition Proxy model. This model is used to learn the state transition mapping relationship. Its network structure design is as follows: First, a feedforward neural network is constructed as a state transition proxy model. The input to this model includes sequential sample data. Adjustment amount of the folding angle compared to the previous moment Afterwards, the network passes through hidden layers, which consist of three fully connected layers. Each fully connected layer is followed by a ReLU activation function to initially extract nonlinear features of the environment. Then, it passes through an output layer containing a linear activation function to output the predicted state. The entire network is determined by parameters. This means that the functional relationship it learns can be written as: .
[0051] During training, mean squared error is used as the loss function to measure the deviation between the predicted state and the true state. ,in For the sample size, This is the i-th training sample. The network parameters are continuously optimized using gradient descent. This allows the model to gradually approximate the dynamics of the real system.
[0052] To balance physical interpretability and data fitting capability, a weighted fusion approach is used to construct the final state transition model. The theoretically optimal state variables obtained based on physical laws are then used. With predicted state We perform weighted fusion to form the optimal state prediction result for the next state. : ; in To integrate the weight parameters and balance the contribution ratios of the physical model and the data-driven model, when... At higher levels, the system relies more on physically interpretable models; when At lower levels, it relies more on data-driven fitting capabilities.
[0053] After the above steps, a high-fidelity simulation environment is obtained. ,in This represents a fusion architecture that combines physical laws and neural network predictions, providing a high-precision, high-stability interactive platform for subsequent policy optimization. The overall process is as follows: Figure 2 As shown.
[0054] S3. Training of PPO-based security constraint deep reinforcement learning strategy S3-1 PPO policy network and value network structure design. A high-fidelity simulation environment constructed based on S2. As input to the training environment, this environment can return the optimal state prediction result in each interaction step. With instant rewards At each time step The intelligent agent executes the following interaction flow: Input the current state vector Policy network output actions Environmental execution Return the optimal state prediction result With rewards Storage state transition samples The data is then transferred to the trajectory cache pool. This process constructs a complete dataset of reinforcement learning interaction trajectories.
[0055] Building upon this, an Actor-Critic framework based on Proximal Policy Optimization (PPO) is constructed to learn the optimal control policy from interaction data, such as... Figure 3 As shown. The policy network is represented by a parameterized policy function. It is composed of neural network parameters The decision is described in the state vector. Take action below The probability distribution.
[0056] Specifically, two sub-models are constructed: a policy network and a value network, which include: The policy network is implemented using a multi-layer fully connected feedforward structure, which consists of an input embedding layer, a feature transformation layer, a gated fusion layer, and a policy output layer. The input embedding layer receives the optimal state prediction result. Linear mapping and normalization are performed to map the original physical quantities to a unified feature space. The feature transformation layer consists of two fully connected layers, and layer normalization is introduced to improve training stability. The gated fusion layer adopts a gated linear unit structure to adaptively weight and fuse environmental and structural features, improving the ability to represent complex sea conditions. Each hidden layer uses the ReLU activation function. The policy output layer adopts a dual-branch structure, with one branch outputting the action mean. The other output motion standard deviation The standard deviation is constrained to a positive value using the Softplus function. Then, the mean is... with standard deviation As parameters, a Gaussian probability distribution of continuous actions is constructed, and control actions are obtained by sampling from this distribution. This forms the policy function: .
[0057] Furthermore, the value network structure is the same as the policy network, consisting of an input layer and two fully connected hidden layers. The output layer outputs a scalar value to represent the state vector. Corresponding state value .in, In the state vector The expected value of future cumulative rewards that can be obtained from this state is indirectly supervised learning through reward signals in reinforcement learning, which is the expected value of future cumulative income.
[0058] S3-2 Construction of the multi-objective safety constraint reward function. During the training of the policy network, the instantaneous rewards collected in S3-1 are used... This transforms the signal into a more stable optimization signal. To this end, a superiority function is introduced to measure the superiority of the current action relative to the average policy.
[0059] First, calculate state value based on value network. Value of the next state Constructing time-series difference error As a fundamental evaluation metric, the return estimate for the current state is obtained. Furthermore, a first-order advantage function is constructed. This advantage function is directly used for subsequent policy updates.
[0060] S3-3 PPO Strategy Update and Pruning Optimization Mechanism. (This refers to obtaining the advantage function.) Then, the PPO algorithm is used to update the policy network. The policy ratio is defined as: ; in, Based on the previous strategy, construct the core objective function for PPO: ; in, The shear coefficient is... This is the shearing function. The policy network parameters are updated by maximizing this objective function. To achieve stable optimization.
[0061] The policy network that eventually converges is obtained. The network can adjust according to the input state. Real-time output of optimal decomposition control action and the corresponding state value .
[0062] S4. Closed-loop control execution and online adaptive optimization S4-1 policy network deployment and real-time status awareness. Optimal control policy obtained from S3 training. and state value The strategy network is deployed in the buoy's local control unit. The system receives state vectors in real time from multiple sensors and processed by S1. , as input for control decisions.
[0063] In each control cycle Inside, the state vector Input high-fidelity simulation environment Then through the policy network Output the average action value With the standard deviation of movement A Gaussian probability distribution is constructed based on the mean and standard deviation of the actions. Control actions are generated by randomly sampling from this distribution. ,and This indicates the adjustment amount of the photovoltaic panel's folding angle. To avoid drastic fluctuations caused by sampling, the sampling results can be smoothed in actual control, i.e., by comparing them with the action at the previous moment. Weighted fusion is performed to obtain smooth control output, thereby improving execution stability.
[0064] S4-2 Deflection Control Execution and Physical System Closed-Loop Response. This refers to the adjustment amount of the photovoltaic panel deflection angle output by the strategy network. The signal is converted into a drive signal, which is then transmitted to the photovoltaic panel folding and unfolding system via a stepper motor actuator to achieve angle adjustment. ; Following the application of control actions, the buoy system generates a new state response under the influence of the real marine environment: the system's next state. This process constitutes a closed-loop control link of strategy output, actuator, and physical system response.
[0065] S4-3 Real-time Feedback Acquisition and Experience Data Construction. During system operation, sensors continuously construct post-execution system response data, including: state vectors. Execute actions Next state Power generation Structural stress Thus, state transition samples in a real-world environment are constructed. It is then placed into the experience replay pool for subsequent online learning and strategy revision.
[0066] S4-4 Online Strategy Update and Model Adaptive Correction. This is to reduce the impact on high-fidelity simulation environments. To address the modeling discrepancies between the actual marine environment and the actual conditions, an online reinforcement learning mechanism is introduced during system operation to incrementally update the policy network.
[0067] Specifically, sample data is extracted from the experience replay pool, and the PPO policy optimization method established in S3 is used to optimize the current policy network. Small-compensation incremental updates are performed to continuously optimize the system based on real-world data. During these updates, the entire training process is not rebuilt; instead, the existing parameters are adjusted using actual runtime data. This involves making adjustments to transition from a simulation-based training strategy to a real-world adaptive strategy. Simultaneously, adjustments are made to the value network. To improve the accuracy of long-term returns assessment in real-world environments, the system is updated synchronously.
[0068] S4-5 Safety Constraints and Stable Operation Mechanism. To ensure long-term stable operation of the system during online control, a multi-layered safety constraint mechanism is introduced. This safety mechanism operates at the control execution layer, rather than the reinforcement learning training process, and is used to impose engineering constraints and real-time corrections on the policy network's output actions.
[0069] Specifically, this includes, firstly, the control actions Set amplitude limits to ensure that they meet the physical constraints of the actuator. Secondly, real-time monitoring of structural stress. When its When this occurs, a safety control mechanism is triggered to forcibly correct the original control action, specifically by scaling the control action proportionally. Force correction, action after correction The original motion input actuator is replaced, reducing the structural stress level and preventing the system from entering dangerous operating conditions. Simultaneously, under extreme sea conditions, the system enters a safety protection model, directly overriding the strategy output, adjusting the photovoltaic panel to a preset safe angle, and suspending strategy control output. Through these mechanisms, engineering-level safety assurance is achieved for the strategy output.
[0070] Through closed-loop control and online optimization processes, the following are ultimately achieved: execution of an adaptive expansion control strategy based on real-time environmental awareness; and a continuously updated and optimized policy network. With value function To achieve long-term stable operation of photovoltaic panels in complex marine environments by maximizing power generation efficiency and optimizing structural safety constraints.
[0071] Simulation experiment: Figure 4 Experimental results show that under the combined effect of multi-source environmental excitation in the first subgraph, the buoy photovoltaic system exhibits obvious nonlinear dynamic response characteristics. Figure 2 The dynamic response curves show that the unfolding angle and angular velocity oscillate continuously with environmental disturbances, indicating that the system is significantly affected by multi-physics coupling. Furthermore, through sub- Figure 3 By constructing a scatter plot relationship between the bending angle and structural stress, it can be observed that the two exhibit obvious nonlinear coupling characteristics and form a dynamic trajectory that evolves over time. This indicates that structural stress is not only affected by the angle but also interacts with environmental loads, verifying the rationality of the constructed stress model. In the sub- Figure 4 In terms of power generation performance, the power output fluctuates periodically with changes in solar radiation intensity and folding angle, and reaches a higher output level when the angle is close to the optimal light-receiving direction, indicating that the folding angle has a significant impact on power generation efficiency.
[0072] Figure 5The simplified simulation results of PPO policy training are presented. The top figure shows the cumulative reward convergence plot, and the bottom figure shows the control angle variation. The results show that as the number of training iterations increases, the cumulative reward shows an overall upward trend and gradually converges, indicating that the constructed policy parameter-based control policy can continuously optimize decision-making performance during continuous interaction with the environment, achieving effective approximation of the objective function. Meanwhile, the evolution of the unfolded angle shows that it fluctuates significantly in the early stages, but gradually stabilizes as the policy is updated, indicating that the agent has learned the optimal action output pattern under different state conditions. In summary, this experiment verifies the effectiveness of the designed reward function and policy update mechanism, enabling a dynamic trade-off between improving power generation performance and suppressing structural risks, ultimately allowing the control policy to converge to a better solution under safety constraints.
[0073] Figure 6 The time-series dynamic response diagrams show the evolution of key variables in the system under closed-loop control and safety constraints. The top diagram is the closed-loop control diagram of the photovoltaic panel's folding angle, the middle diagram is the structural stress and safety threshold diagram, and the bottom diagram is the control input diagram. Experimental results show that the overall system stability is significantly improved after introducing the execution-level safety constraint mechanism. By scaling the control actions proportionally, further stress growth is effectively suppressed. Meanwhile, the photovoltaic panel's folding angle... The evolution process gradually changed from initial random fluctuations to smooth changes, indicating that under the influence of safety constraints, the drastic changes in control input were weakened, and the system's dynamic response became more stable. Looking at the trend of control input changes, its amplitude gradually converged under the dual effects of amplitude limiting constraints and safety corrections, avoiding large adjustments caused by policy randomness.
[0074] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0075] While the above description illustrates specific embodiments of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning, characterized in that: Includes the following processes: S1, the original state vector is constructed by real-time acquisition of marine environment and photovoltaic panel structure status through multi-source sensor array, and a unified state vector is constructed after preprocessing and trigonometric function mapping of direction angle variable; S2 uses a unified state vector as input to perform multi-physics coupled dynamics modeling and formal modeling of Markov decision process. It constructs a multi-objective evaluation mechanism through power generation, structural stress and control energy consumption to form an environmental interaction interface to obtain the theoretical optimal state quantity. Then, it introduces a state transition proxy model to obtain the predicted state. After weighted fusion, it outputs the optimal state prediction result to construct a high-fidelity simulation environment. S3, based on the optimal state prediction results, outputs control actions through the policy network and constructs a state value update value network to perform collaborative optimization among power generation revenue, energy consumption and structural safety constraints, and finally outputs the optimal expanded control action and corresponding state value. S4 deploys the strategy network in the buoy's local control unit, receives the unified state vector in real time, and outputs the optimal unfolded control action by the strategy network.
2. The buoy photovoltaic panel deflection and expansion control method based on deep reinforcement learning as described in claim 1, characterized in that: The original state vector includes: On the environmental side, wave radar acquires effective wave height. With wave cycle Flow velocity and direction meter acquire flow velocity With the angle of flow Wind speed and direction are obtained from the anemometer. with wind angle And radiometers to count solar radiation intensity ; On the structural side, the folding angle of the photovoltaic panel is obtained by using an angle encoder and strain gauges, respectively. angular velocity ; The system uses a fixed sampling period Perform discrete sampling, at the... The original state vector is formed at each moment. .
3. The buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning as described in claim 2, characterized in that: The preprocessing includes time synchronization, linear interpolation and denoising, and normalization; for variable flow direction angles related to the direction angle... and wind direction angle It is processed using triangular mapping.
4. The buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning as described in claim 2, characterized in that: The multiphysics coupling dynamics modeling in S2 specifically involves: First, a marine environmental excitation model is constructed to obtain the wave force expression: ； in The density of seawater, It is the acceleration due to gravity; For the effective wave height, For wave cycles, The current time variable; Meanwhile, wind load With fluid load They are modeled as follows: ； ； in, air density, and The drag coefficient, The area under stress; For wind speed, The ocean current velocity; Three forces through lever arm Converting to torque and then superimposing the directions yields the environmental torque: ； lever arm This represents the equivalent distance between the point of force application on the photovoltaic panel and its axis of rotation. On the structural side, the photovoltaic panel is simplified as a rotating rigid body system, and its dynamic equation is: ； in For rotational inertia, The damping coefficient is... For equivalent stiffness, To control the input torque; Let be the angular acceleration of the photovoltaic panel's folding angle. The angular velocity is the adjustment amount of the photovoltaic panel's folding angle through reinforcement learning feedback. Proceed to the next status update.
5. The buoy photovoltaic panel deflection and expansion control method based on deep reinforcement learning as described in claim 4, characterized in that: The formal modeling of the Markov decision process is specifically as follows: Defined as a quadruple ;in: state space Constructed sequence sample data composition; Action space Adjustment amount for the folding angle of the photovoltaic panel ; State transition function Based on the constructed dynamic equations, for Solve the equation and obtain the solution. ,in This represents the solution obtained from the dynamic equations for calculation. Combination mapping function; and As a physical state transition function, and This is the theoretical state quantity of the photovoltaic panel's folding angle; reward function To guide control strategies to achieve a balance between power generation efficiency, energy consumption, and structural safety: ； in, Photovoltaic power generation capacity, For structural stress response, To control energy consumption, the adjustment of the photovoltaic panel's folding angle is used. With energy consumption coefficient The product is obtained; The safety penalty term is then defined as a piecewise function: when hour, ;when hour, ; , , The importance weights are assigned to power generation revenue, energy consumption penalties, and security constraints.
6. The buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning as described in claim 1, characterized in that: A feedforward neural network is constructed as a state transition proxy model, and the model input includes sequential sample data. Adjustment amount of the folding angle compared to the previous moment Afterwards, the network passes through hidden layers, including three fully connected layers. Each fully connected layer is followed by a ReLU activation function to initially extract non-linear features of the environment. Then, it passes through an output layer containing a linear activation function to output the predicted state. .
7. The buoy photovoltaic panel deflection and expansion control method based on deep reinforcement learning as described in claim 1, characterized in that: In S3, a high-fidelity simulation environment is constructed. As input to the training environment, the environment can return the optimal state prediction result at each step of the interaction. With instant rewards ; at each time step Execute the following interaction flow: Input the current state vector Policy network output actions Environmental execution Return the optimal state prediction result With rewards Storage state transition samples The trajectory cache pool is used to construct an Actor-Critic framework based on near-end policy optimization of PPO, which learns the optimal control policy from interaction data.
8. The buoy photovoltaic panel deflection and expansion control method based on deep reinforcement learning as described in claim 1, characterized in that: The policy network consists of an input embedding layer, a feature transformation layer, a gated fusion layer, and a policy output layer. The input embedding layer receives the optimal state prediction result. Linear mapping and normalization are performed to map the original physical quantities to a unified feature space; The feature transformation layer consists of two fully connected layers, and layer normalization is introduced to improve training stability. The gated fusion layer adopts a gated linear unit structure to adaptively weight and fuse environmental and structural features; the policy output layer adopts a dual-branch structure, with one branch outputting the average action value. The other output motion standard deviation The standard deviation is constrained to a positive value by the Softplus function; Then, the mean with standard deviation As parameters, a Gaussian probability distribution of continuous actions is constructed, and control actions are obtained by sampling from this distribution. This forms the policy function: ; The value network structure is the same as the policy network, consisting of an input layer and two fully connected hidden layers. The output layer outputs a scalar value to represent the state vector. Corresponding state value .
9. The buoy photovoltaic panel folding and unfolding control method based on deep reinforcement learning as described in claim 1, characterized in that: It also includes an online reinforcement learning mechanism for incremental updates to the policy network: extracting sample data from the experience replay pool and optimizing the current policy network based on the established PPO policy optimization method. Small-compensation incremental updates are performed to continuously optimize the system based on real-world data. During these updates, actual operational data is used to refine the original parameters. Make corrections; at the same time, adjust the value network. Perform synchronized updates.
10. The buoy photovoltaic panel deflection and expansion control method based on deep reinforcement learning as described in claim 1, characterized in that: It also includes a multi-layered security constraint mechanism: First, control actions Set amplitude limits to ensure that they meet the physical constraints of the actuator. ; Secondly, real-time monitoring of structural stress When its When this occurs, a safety control mechanism is triggered, forcibly correcting the original control action by scaling it proportionally. Force correction, action after correction The original action input actuator is replaced, reducing the stress level on the structure and avoiding dangerous working conditions. At the same time, under extreme sea conditions, the system enters the safety protection model, directly overriding the strategy output, adjusting the photovoltaic panel to the preset safe angle, and suspending the strategy control output.