A Multidisciplinary Optimization Design Method for Underwater Robots Based on Deep Reinforcement Learning
By constructing a multidisciplinary optimization model through deep reinforcement learning, and combining the Kriging agent model and Markov decision process, the problems of disciplinary coupling and computational cost in the multidisciplinary optimization design of underwater robots are solved, realizing efficient and adaptive optimization design and reducing engineering costs and time.
Patent Information
- Application Number
- CN202611261200.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies for multidisciplinary optimization design of underwater robots suffer from problems such as insufficient consideration of interdisciplinary coupling, high computational costs, lack of adaptability to working conditions in optimization results, and reliance on commercial software by traditional platforms. These issues result in optimization results that are not feasible in engineering and excessive computation time.
A multidisciplinary optimization model based on deep reinforcement learning is constructed, which includes hydrodynamics, structural mechanics, energy dynamics, maneuverability, and overall layout. A Kriging surrogate model is used to replace high-fidelity simulation, and the model is modeled as a Markov decision process. A flexible action evaluation algorithm is used to train the policy network and adjust the reward function weights in real time to adapt to different task conditions.
It achieves engineering feasibility for multidisciplinary optimization problems, reduces computational costs, obtains a design scheme with comprehensive performance optimization, and has the ability to adapt to operating conditions, thus reducing dependence on commercial software.
Smart Images

Figure CN123116799A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of underwater robot overall design and optimization technology, specifically involving a multidisciplinary optimization design method for underwater robots based on deep reinforcement learning. Background Technology
[0002] The overall performance of an autonomous underwater vehicle (AUV) depends on the collaborative design of multiple disciplines, including hydrodynamics, structural mechanics, energy and power, maneuverability, and overall layout. These disciplines are strongly coupled and mutually restrictive.
[0003] Currently, multidisciplinary optimization design for AUVs mainly falls into two categories: multidisciplinary design optimization methods based on traditional optimization algorithms and optimization methods based on deep reinforcement learning. The first category, based on traditional optimization algorithms, models the multidisciplinary optimization problem as a single-objective or multi-objective optimization problem, employing evolutionary algorithms such as genetic algorithms, particle swarm optimization, or non-dominated sorting genetic algorithms for global search. Its typical technical route involves: establishing a parametric shape model of the AUV (e.g., Myring curves); generating initial sample points through experimental design methods; using high-fidelity simulation tools such as Computational Fluid Dynamics (CFD) and Finite Element Analysis (FEA) to calculate the performance indicators of the sample points; constructing a surrogate model (e.g., the Kriging model) to approximate the high-fidelity simulation; and finally, using an evolutionary algorithm to iteratively search for the optimal solution on the surrogate model. Currently, multidisciplinary optimization software platforms, such as Isight, have achieved multidisciplinary collaborative optimization by integrating CAD / CAE tools, experimental design, surrogate models, and evolutionary algorithms. The second category is based on deep reinforcement learning (DRL) methods, which learn the optimal strategy through trial and error by interacting with the environment. They have stronger sample utilization efficiency and continuous decision-making ability, and have been gradually applied to underwater robot path planning, low-level control and single-discipline optimization in recent years.
[0004] However, the aforementioned existing technologies still have the following shortcomings: First, the existing methods based on DRL (Digital Dynamics Reduction) do not adequately consider interdisciplinary coupling, resulting in poor engineering feasibility of the optimized results. Most existing methods only address a single discipline or consider simple coupling between two disciplines, failing to incorporate the coupling constraints of multiple disciplines such as structural mechanics, energy dynamics, maneuverability, and overall layout into a unified optimization framework. However, there are typical constraints between the disciplines of AUVs—hydrodynamics pursues a slender shape to reduce drag, structural mechanics requires a short and sturdy shape to improve pressure resistance and efficiency, and energy science pursues large-capacity batteries to extend range but increases structural mass and overall layout complexity. Existing methods ignore these coupling relationships, and the low-drag shape obtained through optimization may lead to insufficient internal volume or an unreasonable center of gravity position, making the optimized results often infeasible in practical engineering.
[0005] Second, the high computational cost severely restricts the application of DRL. DRL training requires tens of thousands to millions of interactions. If each interaction involves calling CFD (which takes hours per interaction) and FEA (which takes tens of minutes per interaction), the total computation time can reach months or even years, making it impractical in engineering.
[0006] Third, traditional optimization methods lack adaptability to operating conditions, obtaining a static optimal set of design parameters that cannot be dynamically adjusted according to changes in the mission profile or marine environment. At the same time, commercial platforms such as Isight rely on third-party software licenses, making it difficult to reproduce and share the process.
[0007] Fourth, traditional multidisciplinary optimization methods usually pre-set the weight coefficients of each discipline when dealing with interdisciplinary conflicts. This is highly subjective and can only obtain fixed solutions. It lacks an adaptive multidisciplinary conflict balancing mechanism. Although DRL naturally has the ability to learn strategies, existing research has not yet effectively applied its advantages to the field of multidisciplinary optimization design of AUV.
[0008] Therefore, there is an urgent need to provide a multidisciplinary optimization design method for underwater robots based on deep reinforcement learning to address the shortcomings of existing technologies. Summary of the Invention
[0009] To address the problems in multidisciplinary optimization of underwater robots in related technologies, such as insufficient consideration of disciplinary coupling, high computational cost of DRL training, lack of adaptability to different operating conditions in optimization results, and reliance on commercial software by traditional platforms, this invention provides a multidisciplinary optimization design method based on deep reinforcement learning. This method establishes a coupled optimization model encompassing five disciplines: hydrodynamics, structural mechanics, energy and power, maneuverability, and overall layout. It employs a Kriging surrogate model to replace high-fidelity simulation to reduce DRL training costs, models the multidisciplinary optimization problem as a Markov decision process, and trains the policy network using a flexible action evaluation algorithm. After training, the reward function weight coefficients can be adjusted to adapt to different task conditions in real time.
[0010] The technical solution of this invention is implemented as follows: a multidisciplinary optimization design method for underwater robots based on deep reinforcement learning, comprising the following steps: S1. Construct a multidisciplinary optimization model, use Myring curves to parametrically describe the shape of the underwater robot body, and define design variables; establish performance evaluation models for each discipline. S2. Using experimental design methods, sample points are generated in the design space. For each sample point, a high-fidelity simulation tool is called to calculate the true value of the performance index. The Kriging model is used to establish the mapping from the design variables to each performance index. Independent Kriging models are trained for multiple performance indices respectively. S3. Model the multidisciplinary optimization problem as a Markov decision process, define the state space, action space, and state update rules; define the reward function; S4. Construct an intelligent agent using a deep reinforcement learning algorithm, build an Actor network and a Critic network, initialize the experience replay pool, and iteratively execute policy sampling, agent model prediction, reward calculation and network parameter update until the Actor network converges. S5. Sample from the converged Actor network to obtain candidate optimal design parameters, verify the candidate design using high-fidelity simulation, and output the final optimization result.
[0011] This invention significantly reduces the computational cost of deep reinforcement learning training by constructing a multidisciplinary coupled optimization model and using a Kriging surrogate model to replace high-fidelity simulation for performance prediction, enabling the DRL method to be practically applied to the multidisciplinary optimization design of underwater robots. At the same time, by modeling the multidisciplinary optimization problem as a Markov decision process and training an Actor network, an integrated design scheme that takes into account the performance indicators of multiple disciplines is obtained, improving the overall performance of the underwater robot design.
[0012] As a further improvement to the above scheme, the multidisciplinary field includes hydrodynamics, structural mechanics, energy dynamics, maneuverability, and overall layout.
[0013] By specifically limiting the disciplines involved in the multidisciplinary optimization model to five disciplines—hydrodynamics, structural mechanics, energy and power, maneuverability, and overall layout—the scope of disciplines that need to be comprehensively considered during the optimization process is clarified, ensuring that the coupling relationships between various disciplines are incorporated into a unified optimization framework.
[0014] As a further improvement to the above solution, the multiple performance indicators include casing mass, battery mass, pressure resistance depth, and turning radius.
[0015] By specifying performance indicators as casing mass, battery mass, pressure resistance depth, and turning radius, a complete performance evaluation basis is provided for multidisciplinary coupled optimization.
[0016] As a further improvement to the above scheme, in S3, the state vector in the state space is composed of the following design variables: slenderness ratio. The ratio of bow length to total length The ratio of the length of the parallel body to the total length Bow shape index n, stern shape index m, pressure hull thickness t, and the ratio of rudder area to reference area. ; The state space is defined as follows: ; The action space is defined as a 7-dimensional continuous action vector, representing the adjustment amount of each design variable: ; in, For slenderness ratio adjustment; For the bow length ratio adjustment; This is the adjustment amount for the parallel body ratio; Adjustments to the bow shape index; Adjustment amount for stern shape index; This is the thickness adjustment amount; This refers to the adjustment amount for the rudder area ratio; The state update rule is as follows: ; in, This is the updated state.
[0017] By quantifying the design variables and their adjustments into state space and action space respectively, and establishing state update rules, the agent can gradually search for the optimal design solution in the continuous space, thereby transforming the multidisciplinary optimization problem into a sequential decision problem.
[0018] As a further improvement to the above scheme, the reward function is defined as: ; in, For battery life; For pressure resistance depth; Radius of gyration; , These are reference values for each indicator; , , These are the weighting coefficients; To constrain penalty items.
[0019] By converting the three core indicators of endurance, pressure resistance depth, and turning radius into a single scalar reward value through a weighted summation, and incorporating constraint violations into the same reward function as penalty terms, multi-objective, multi-constraint, and multi-disciplinary optimization problems can be solved under a unified evaluation framework.
[0020] As a further improvement to the above scheme, the constraint penalty term is defined as follows: ; in, For the maximum equivalent stress, For allowable stress, For the material's yield strength, For safety reasons, , This is the penalty coefficient.
[0021] By quantifying stress constraint violations and design variable boundary violations as penalty terms and incorporating them into the reward function, the constraint satisfaction of the design scheme can be directly reflected in the reward value.
[0022] As a further improvement to the above scheme, the iterative execution of policy sampling, surrogate model prediction, reward calculation, and network parameter update until the Actor network converges is specifically as follows: The design parameters are updated based on the current strategy sampling action, and the updated design parameters are then subject to boundary clipping. The proxy model is invoked to predict the performance metrics corresponding to the updated design parameters; Calculate the reward value based on the reward function; Store the experience in the experience replay pool; The network parameters are updated by sampling experience from the experience replay pool; Repeat the above steps until the Actor network converges.
[0023] By establishing a closed-loop interactive process of policy sampling, agent model prediction, reward calculation, and network parameter update, the agent can autonomously optimize design parameters through continuous interaction with the agent model until the Actor network converges.
[0024] As a further improvement to the above scheme, the high-fidelity simulation includes: computational fluid dynamics simulation, finite element analysis simulation, and maneuverability simulation based on Fossen's equations of motion.
[0025] By limiting high-fidelity simulation to computational fluid dynamics simulation, finite element analysis simulation, and maneuverability simulation based on Fossen's equations of motion, the performance evaluation of the three disciplines of hydrodynamics, structural mechanics, and maneuverability can all be supported by data provided by the corresponding high-fidelity simulation tools.
[0026] As a further improvement to the above scheme, the network parameter update includes updating the Critic network parameters, updating the Actor network parameters, and updating the parameters of each target network.
[0027] By simultaneously updating the parameters of the Critic network, Actor network, and target network, the parameters of each network are trained and optimized during the interactive iteration process, thereby supporting the stable convergence of the agent's policy.
[0028] As a further improvement to the above scheme, in step S2, a coefficient of determination is used. Evaluate the prediction accuracy of each Kriging model. If the coefficient of determination of any model is... If the preset accuracy threshold is not met, the model is retrained by adding sample points until the prediction accuracy of all models meets the preset accuracy threshold.
[0029] The prediction accuracy of the Kriging model is quantitatively evaluated using the coefficient of determination R², and additional samples are added for retraining when the accuracy does not meet the preset threshold, forming an accuracy-driven closed-loop feedback mechanism that ensures that the prediction accuracy of each surrogate model can reach the preset standard.
[0030] Compared with traditional processing methods, the present invention has the following advantages: (1) This invention constructs a multidisciplinary optimization model that includes four core disciplines: hydrodynamics, structural mechanics, energy and power, and maneuverability. It incorporates key design variables such as slenderness ratio, bow and stern shape index, hull thickness, and rudder area ratio into a unified state space, and simultaneously optimizes three core indicators—endurance, pressure resistance depth, and turning radius—through a reward function. The interrelationships between the disciplines are naturally incorporated into the optimization framework, making the final design scheme truly feasible in engineering.
[0031] (2) Although traditional multidisciplinary optimization methods based on platforms such as Isight employ surrogate model technology, their built-in genetic algorithms sample independently in each iteration, without utilizing historical data knowledge, still requiring thousands of iterations and resulting in high computational costs. This invention adopts an offline training approach for the Kriging surrogate model: firstly, a sample set is generated through 200-500 high-fidelity simulations (CFD / FEA / manipulation simulations) to train a high-precision (R2>0.9) surrogate model; during DRL training, the environment only needs to call the surrogate model to predict performance indicators, with each prediction taking only milliseconds. This makes DRL training, which originally required tens of thousands of high-fidelity simulations, engineering feasible.
[0032] (3) This invention employs the SAC deep reinforcement learning algorithm, whose output is a policy network rather than a single-point solution. The trained policy network can dynamically output the corresponding optimal design parameters according to different task requirements and environmental parameters (adjusted by the weight coefficients in the reward function). By adjusting the weight coefficients in the reward function, this method can adapt to different task preferences: the endurance weight can be increased for long-endurance reconnaissance missions, the pressure resistance weight can be increased for deep-sea exploration missions, and the maneuverability weight can be increased for shallow-water agile operations, without the need for retraining.
[0033] (4) The technical solution of this invention can be implemented based on the Python open-source ecosystem: Myring curve parametric modeling, Latin hypercube sampling, Kriging surrogate model (using SMT or scikit-learn library), SAC deep reinforcement learning algorithm (using Stable-Baselines3 or PyTorch), Fossen motion equation manipulation simulation, etc. can all be completed under the open-source framework. The overall optimization process does not rely on any commercial software license, the code is publicly available, reproducible, and can be further developed, which significantly reduces the technical application threshold and promotion cost. Attached Figure Description
[0034] Figure 1 This is an overall flowchart of an embodiment of the present invention; Figure 2 This is a diagram illustrating the multidisciplinary coupling relationships in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the proxy model construction process in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the reward function calculation process according to an embodiment of the present invention. Figure 5 This is a diagram of the DRL training framework according to an embodiment of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] Example 1 like Figure 1 As shown, this embodiment provides a multidisciplinary optimization design method for underwater robots based on deep reinforcement learning, including the following steps: Step S1: Construction of a multidisciplinary optimization model. A multidisciplinary optimization model is constructed, encompassing hydrodynamics, structural mechanics, energy dynamics, maneuverability, and overall layout. These disciplines are coupled through shared external and internal layout parameters.
[0037] The Myring curve is used to parametrically describe the shape of the AUV body. The Myring curve is defined by the following piecewise function: ; in, The maximum diameter is expressed in meters (m). Total length, in meters (m); The bow length is in meters (m). The length of the parallel body is given in meters. The stern length is in meters; and satisfies the following conditions: ; and These are the shape indices for the bow and stern, respectively. In this embodiment, the value ranges from 1.5 to 3.0, with larger values indicating sharper shapes.
[0038] In this embodiment, the design variables are the states of the DRL agent, which have a total of 7 dimensions, as shown in Table 1.
[0039] Table 1 Given known quantities, as shown in Table 2.
[0040] Table 2 Performance evaluation models for each discipline are established separately. In hydrodynamics, during cruise, thrust and drag are balanced: ; in, Speed, measured in m / s; This represents the wetted surface area, in m². The coefficient of frictional resistance. Let Reynolds number be 1. The shape factor is given by the thrust. Inverse speed Then calculate the power requirement: .
[0041] In structural mechanics, the circumferential stress of the parallel midbody segment is calculated using the thin-shell cylindrical shell theory: ; ; in Working depth (m). Standard atmospheric pressure. Shell mass: ; Strength constraints: .
[0042] In the field of energy dynamics, battery mass is derived from the geometry of the pressure chamber. (Pressure chamber inner diameter) inner cross-sectional area Battery cross-sectional area Battery volume Battery quality Total mass Battery life: ; In maneuverability science, the slender body theory is used, and the turning radius is solved based on Fossen's equations of motion. The turning radius is affected by the slenderness ratio, the rudder area ratio, and the total mass. = .
[0043] like Figure 2 As shown, between hydrodynamics and structural mechanics: a slender shape is beneficial for drag reduction but reduces pressure resistance efficiency, while a short and stubby shape is beneficial for pressure resistance but increases drag; between hydrodynamics and energy dynamics: drag affects power demand and thus endurance; between structural mechanics and energy dynamics: the geometry of the pressure tank determines the available battery space, affecting battery mass and endurance; between maneuverability and structural mechanics and energy dynamics: total mass (including shell mass, battery mass, and equipment mass) affects the turning radius through inertia; between maneuverability and hydrodynamics: the slenderness ratio and rudder area ratio jointly affect the turning radius.
[0044] Step S2: Construction and training of the proxy model.
[0045] like Figure 3 As shown, in this embodiment, a Kriging surrogate model is used instead of high-fidelity simulation. First, optimal Latin hypercube sampling is used to generate [the model] within the design space. Each sample point contains a set of design variables. .
[0046] For each sample point, the following high-fidelity simulation tools are called sequentially to calculate the true value of the performance index: CFD Simulation: Calculating Total Drag under a Given Shape and wet surface area ; FEA Simulation: Calculating Shell Mass and pressure resistance depth ; Maneuverability Simulation: Calculating the Radius of Gyratory Based on Fossen's Equations of Motion ; By mapping the simulation results one-to-one with the design variables, a sample set is constructed: .
[0047] The Kriging model is used to establish the mapping between design variables and various performance indicators. The Kriging model is expressed as: ; in, For the regression trend term, With a mean of 0 and a variance of Gaussian process.
[0048] Regarding the shell mass Battery quality Pressure resistance depth Radius of gyration Independent Kriging models were trained using four performance metrics. The coefficient of determination was employed. To evaluate the accuracy of the proxy model, each model is required to... If the value is >0.9, otherwise supplement the samples and retrain.
[0049] Step S3: Markov Decision Process Modeling. The multidisciplinary optimization problem is modeled as a Markov decision process, defining the state space, action space, and state update rules; a reward function is also defined.
[0050] State vector It includes 7 design variables: ; Action vectors This represents the adjustment amount of each design variable, with a total of 7 dimensions of continuous action: ; The status update rules are as follows: ; in, This is the updated state.
[0051] After the update, boundary clipping is performed on each state variable to ensure that it does not exceed its own value range.
[0052] like Figure 4 As shown, this embodiment uses three core indicators to evaluate the merits of the design scheme: battery life. Pressure resistance depth Radius of gyration The reward function is defined as: ; in, , These are reference values for each indicator, used for normalization; , , For the weighting coefficients, satisfying It can be adjusted according to the needs of the task.
[0053] Constraints and penalties Defined as: ; in, For the maximum equivalent stress, For allowable stress, 10. 5 is the penalty coefficient.
[0054] Step S4: Construction and training of deep reinforcement learning agents.
[0055] like Figure 5 As shown, in this embodiment, the Soft Actor-Critic (SAC) algorithm is used. SAC encourages the agent to explore fully by introducing policy entropy into the reward maximization objective, and has good performance and stability in continuous action spaces.
[0056] The objective function of SAC is: ; in, For policy entropy, This is the temperature coefficient.
[0057] Both the Actor and Critic networks employ a multilayer perceptron structure. The Actor network has an input layer dimension of 7, corresponding to the state space dimension, two hidden layers with 256 neurons each, and an output layer dimension of 14, corresponding to the seven action means and seven log-standard deviations. The Critic network is a dual-network structure, with each Critic network having an input layer dimension of 14, corresponding to the concatenation dimension of the state and action vectors, two hidden layers with 256 neurons each, and an output layer dimension of 1.
[0058] The training hyperparameters for this embodiment are shown in Table 3.
[0059] Table 3 The training process is as follows: First, initialize the Actor network. Two Critic networks , and the corresponding target network , Initialize the experience replay pool Randomly initialize initial design parameters The S2 proxy model is invoked to calculate initial performance metrics. Subsequently, for each step... =0,1,2,…, Repeat the following steps until the Actor network converges: Sampling actions based on the current strategy: Update design parameters: And perform boundary clipping; Call the S2 proxy model to predict Corresponding performance indicators ( ); Based on the given thrust Thrust Reverse Speed Calculate battery life ; Calculate reward value ; Experience Store in the playback pool ; From the playback pool Sampling and updating network parameters includes updating the Critic network: minimizing the Bellman residual; updating the Actor network: maximizing the entropy-regularized cumulative reward; and soft updating the target network. ; Repeat the above steps until the Actor network converges.
[0060] Step S5: Optimal Design Output. Sample from the converged policy network to obtain multiple sets of candidate optimal design parameters. The three core metrics (range, pressure resistance, and turning radius) are calculated to select the Pareto optimal solution set. A high-fidelity simulation is then run to verify the candidate design, and the final optimization results are output.
[0061] Example 2 This embodiment takes a typical medium-sized autonomous underwater vehicle as the optimization object. Based on the technical solution described in Embodiment 1, it details the complete implementation process and optimization effect of the method of the present invention with specific parameters.
[0062] The known quantities are shown in Table 4.
[0063] Table 4 The initial values and ranges of the design variables are set as shown in Table 5.
[0064] Table 5 Optimal Latin hypercube sampling (generated within the design space) is employed. There are 10 sample points, each containing a set of 7-dimensional design variables: ; The sampling strategy ensures that sample points are evenly distributed throughout the design space, avoiding local clustering. 300 sample points cover the entire design space with aspect ratios ranging from 4 to 10, shell thicknesses from 4 mm to 20 mm, and rudder area ratios from 0.05 to 0.20.
[0065] For each sample point, the following high-fidelity simulation tools are called sequentially to calculate the true value of the performance index: CFD simulation: Using a solver based on the Reynolds-averaged Navier-Stokes equations, the total drag under a given shape is calculated. and wet surface area The computational domain uses a hemispherical inlet and a cylindrical outlet, with a mesh size of approximately 2 million. The SST turbulence model is selected. The model takes about 2 hours to complete a single simulation.
[0066] FEA Simulation: A pressure hull model was established using finite element analysis software. A hydrostatic pressure (approximately 2.0 MPa) corresponding to a depth of 200 m was applied, with the constraint condition being simply supported at both ends. The mass of the hull was calculated. and maximum equivalent stress Each analysis takes approximately 0.5 hours.
[0067] Maneuverability Simulation: A Python simulation program was written based on Fossen's equations of motion. External parameters and total mass were input, a fixed rudder angle of 10 degrees was applied, and the equations of motion for the three degrees of freedom in the horizontal plane were solved to calculate the steady-state turning radius. Each simulation takes approximately 0.1 hours.
[0068] The total simulation time for 300 sample points was approximately 780 hours, or about 32.5 days. This is a one-time offline cost; the simulation can be reused after completion.
[0069] The Kriging model is used to establish a mapping from 7 design variables to 4 performance metrics. The Kriging model is expressed as: ; in, For the regression trend term, With a mean of 0 and a variance of Gaussian process.
[0070] Using 240 samples as the training set and 60 samples as the test set, the shell quality was evaluated. Battery quality Pressure resistance depth Radius of gyration Train independent Kriging models. Use the coefficient of determination. The accuracy of the proxy model was evaluated, as shown in Table 6.
[0071] Table 6 All proxy models This meets the usage requirements. For areas with low prediction accuracy, additional sample points can be added for retraining to improve model accuracy.
[0072] In this embodiment, a balanced task preference is adopted, and the reward function weights are set as shown in Table 7.
[0073] Table 7 Constraint penalty coefficient: , .
[0074] The complete expression for the reward function is: .
[0075] In this embodiment, the state vector It includes 7 design variables: ; Action vectors This represents the adjustment amount of each design variable, with a total of 7 dimensions of continuous action: ; In this embodiment, the value range of each action component is shown in Table 8.
[0076] Table 8 The status update rules are as follows: ; After the update, boundary trimming is performed on each state variable to ensure that the value does not exceed the range of the design variables.
[0077] In this embodiment, the SAC (Soft Actor-Critic) algorithm is used, and the hyperparameter configuration is shown in Table 9.
[0078] Table 9 In this embodiment, both the Actor network and the Critic network employ a multilayer perceptron structure: the Actor network has an input layer dimension of 7, corresponding to the state space dimension, two hidden layers with 256 neurons each, and an output layer dimension of 14, corresponding to the 7 action means and 7 log standard deviations. The Critic network is a dual-network structure, with each Critic network having an input layer dimension of 14, corresponding to the concatenation dimension of the state vector and action vector, two hidden layers with 256 neurons each, and an output layer dimension of 1.
[0079] In this embodiment, the average reward and the average of the three core indicators are recorded every 5000 steps, as shown in Table 10.
[0080] Table 10 When you reach 80,000 steps, the reward value fluctuates less than 5,000 consecutive steps. Furthermore, all three core indicators have stabilized, indicating that the decision strategy network has converged.
[0081] In this embodiment, the total training time was approximately 6 hours. The hardware configuration was a single workstation with an Intel Xeon Gold 6248 CPU, an NVIDIA RTX 3080 GPU, and 64GB of memory.
[0082] Once trained, the policy network can be saved and used for subsequent design parameter generation and adaptive optimization under operating conditions without retraining.
[0083] Based on the trained policy network, the optimal design parameters are obtained, as shown in Table 11.
[0084] Table 11 The performance comparison before and after optimization is shown in Table 12.
[0085] Table 12 A control group was set up, and the NSGA-II algorithm was used to perform multi-objective optimization on the same surrogate model. The population size was 100, the iteration was 200 generations, and the total number of evaluations was 20,000. The comparison results are shown in Table 13.
[0086] Table 13 The optimized design parameters were re-substituted into CFD / FEA / manipulation simulation for verification. The comparison between the surrogate model predictions and the high-fidelity simulation values is shown in Table 14.
[0087] Table 14 As can be seen, the prediction errors of all performance indicators are within 3%, and the accuracy of the surrogate model meets the engineering requirements.
[0088] In this embodiment, the optimal design parameters for different task preferences can be obtained simply by adjusting the weight coefficients of the reward function without retraining. The following uses four task conditions to verify the adaptive capability of the policy network trained by the method in this embodiment. The definitions of the four task conditions are shown in Table 15.
[0089] Table 15 Using the same trained policy network, only the weight coefficients of the reward function were changed to obtain the optimal design parameters under different operating conditions, as shown in Table 16.
[0090] Table 16 The performance under different operating conditions is shown in Table 17.
[0091] Table 17 From the above results, we can obtain the correspondence between operating condition preferences and design parameters: Range Priority (Operating Condition A): Adopting a slender shape (length-to-slenderness ratio 8.2), thin shell (5.5mm), and small rudder (0.10) to reduce drag and weight and improve range; however, the pressure resistance is reduced (185m) and the turning radius is increased (5.0m). Pressure resistance priority (condition B): adopts a short and thick shape (length-to-thickness ratio of 6.5) and a thick shell (8.5mm) to enhance structural strength; the pressure resistance depth reaches 305m, but the endurance is shortened to 10.5h and the weight increases; Maneuverability Priority (Operating Condition C): Employs a medium aspect ratio (7.1) and a large rudder (0.16) to enhance turning capability; the turning radius is only 3.8m, but both range and pressure resistance are compromised. Balanced type (condition D): Achieving a balance among various disciplines, with all three indicators at a moderate level.
[0092] Traditional multi-objective optimization methods (such as NSGA-II combined with a surrogate model) require multiple independent optimizations to obtain design points under different task preferences. Each optimization requires rerunning the algorithm, taking approximately 16 hours (about 4 hours per run) to obtain designs for four different operating conditions. The policy network of this invention requires only one training run (approximately 6 hours) and can output optimal design parameters for different operating conditions in real time by adjusting the reward function weights. In contrast, traditional methods require rerunning optimization when operating conditions change, resulting in discrete design results, independent optimization for each operating condition, and non-reusable design knowledge. The method of this invention provides real-time output for operating condition changes, the design parameters output by the policy network are continuously adjustable in the weight space, and the trained policy network can be reused, demonstrating significant advantages in computational efficiency and design knowledge reuse.
[0093] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.
Claims
1. A multidisciplinary optimization design method for underwater robots based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct a multidisciplinary optimization model, use Myring curves to parametrically describe the shape of the underwater robot body, and define design variables; establish performance evaluation models for each discipline. S2. Using experimental design methods, sample points are generated in the design space. For each sample point, a high-fidelity simulation tool is called to calculate the true value of the performance index. The Kriging model is used to establish the mapping from the design variables to each performance index. Independent Kriging models are trained for multiple performance indices respectively. S3. Model the multidisciplinary optimization problem as a Markov decision process, define the state space, action space, and state update rules; define the reward function; S4. Construct an intelligent agent using a deep reinforcement learning algorithm, build an Actor network and a Critic network, initialize the experience replay pool, and iteratively execute policy sampling, agent model prediction, reward calculation and network parameter update until the Actor network converges. S5. Sample from the converged Actor network to obtain candidate optimal design parameters, verify the candidate design using high-fidelity simulation, and output the final optimization result.
2. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The multidisciplinary field includes hydrodynamics, structural mechanics, energy dynamics, maneuverability, and overall layout.
3. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The aforementioned performance indicators include casing weight, battery weight, pressure resistance depth, and turning radius.
4. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the state vector in the state space is composed of the following design variables: slenderness ratio. The ratio of bow length to total length The ratio of the length of the parallel body to the total length Bow shape index n, stern shape index m, pressure hull thickness t, and the ratio of rudder area to reference area. ; The state space is defined as follows: ; The action space is defined as a 7-dimensional continuous action vector, representing the adjustment amount of each design variable: ; in, For slenderness ratio adjustment; This is the adjustment amount for the bow length ratio; This is the adjustment amount for the parallel body ratio; Adjustments to the bow shape index; Adjustment amount for stern shape index; This is the thickness adjustment amount; This refers to the adjustment amount for the rudder area ratio; The state update rule is as follows: ; in, This is the updated state.
5. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The reward function is defined as follows: ; in, For battery life; For pressure resistance depth; Radius of gyration; , These are reference values for each indicator; , , These are the weighting coefficients; To constrain penalty items.
6. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 5, characterized in that, The constraint penalty term is defined as follows: ; in, For the maximum equivalent stress, For allowable stress, For the material's yield strength, For safety reasons, , This is the penalty coefficient.
7. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The process of cyclically executing policy sampling, surrogate model prediction, reward calculation, and network parameter updates until the Actor network converges is as follows: The design parameters are updated based on the current strategy sampling action, and the updated design parameters are then subject to boundary clipping. The proxy model is invoked to predict the performance metrics corresponding to the updated design parameters; Calculate the reward value based on the reward function; Store the experience in the experience replay pool; The network parameters are updated by sampling experience from the experience replay pool; Repeat the above steps until the Actor network converges.
8. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The high-fidelity simulations include: computational fluid dynamics simulations, finite element analysis simulations, and maneuverability simulations based on Fossen's equations of motion.
9. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, The network parameter update includes updating the Critic network parameters, updating the Actor network parameters, and updating the parameters of each target network.
10. The multidisciplinary optimization design method for underwater robots based on deep reinforcement learning according to claim 1, characterized in that, In S2, the coefficient of determination is used. Evaluate the prediction accuracy of each Kriging model. If the coefficient of determination of any model is... If the preset accuracy threshold is not met, additional sample points are added and the model is retrained until the prediction accuracy of all models meets the preset accuracy threshold.
