Ocean platform pipeline laying method based on double-agent reinforcement learning fusion A satellite
By integrating dual-agent reinforcement learning with the A* algorithm, the pipeline laying of marine platforms is optimized, solving the problems of insufficient constraint adaptability, convergence efficiency and large-scale spatial adaptability of traditional design methods, and realizing efficient and safe pipeline layout.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNIV OF PETROLEUM (EAST CHINA)
- Filing Date
- 2026-04-22
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional offshore platform pipeline design methods are difficult to meet the engineering requirements of high safety, high manufacturability, and low cost. Existing algorithms have significant limitations in terms of constraint adaptability, convergence efficiency, and large-scale spatial adaptability.
A pipeline laying method for marine platforms based on dual-agent reinforcement learning and A-Star algorithm is adopted. Through three-dimensional grid modeling, dual-agent collaborative training and experience playback mechanism, combined with Q-Learning algorithm and A-Star algorithm, the pipeline layout is optimized to achieve multi-objective optimization of path feasibility, shortest path and fewest turns.
It improves design efficiency and quality, reduces labor costs and design risks, generates better pipeline solutions, improves training efficiency, reduces repeated debugging during the design process, and lowers construction and maintenance costs and platform operation risks.
Smart Images
Figure CN122065480A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of marine platform pipeline design technology, specifically to a marine platform pipeline laying method based on dual-agent reinforcement learning fusion of satellite A. Background Technology
[0002] As the core infrastructure for offshore oil and gas development, the pipeline system of offshore platforms undertakes key functions such as oil and gas transportation, cooling circulation, fire protection, and hydraulic control, serving as the "vascular network" for the platform's safe and stable operation. With the advancement of offshore oil and gas development into deep and ultra-deep water, platform compartments are becoming increasingly compact, leading to a surge in the types and number of pipelines (such as high-pressure oil and gas pipes, cryogenic cooling pipes, and corrosion-resistant fire-fighting pipes). Traditional pipeline design methods are no longer sufficient to meet the engineering requirements of "high safety, high manufacturability, and low cost," necessitating the introduction of intelligent algorithms to overcome design bottlenecks.
[0003] Intelligent design of pipeline laying for offshore platforms mainly relies on two types of technical approaches: deterministic path planning algorithms and heuristic optimization algorithms. Both types of technologies have significant limitations in practical engineering applications. Deterministic algorithms, represented by Dijkstra's algorithm and A* algorithm, are based on the core logic of searching for the optimal path from the starting point to the destination based on a "cost function." They have stability advantages in simple, unconstrained scenarios (such as short path planning in open spaces), but they suffer from three major shortcomings in offshore platform pipeline design: constraint adaptability, low space utilization, and poor efficiency in large-scale spaces. Swarm intelligence algorithms, represented by ant colony algorithms and genetic algorithms, suffer from slow convergence. Reinforcement learning algorithms, represented by single-agent Q-Learning, have strong environmental interaction capabilities, but they have inherent defects such as insufficient exploration in large-scale spaces and difficulty in guaranteeing path optimality. Summary of the Invention
[0004] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and provide a method for pipeline laying on marine platforms based on dual-agent reinforcement learning fusion with the A-Star algorithm. The method is designed based on the Q-Learning algorithm of reinforcement learning and the A-Star algorithm. Through the interaction between the agent and the layout environment, the agent can automatically explore different pipeline layout schemes and continuously learn and optimize. Based on the experience of the agent, the A-Star algorithm evaluates the advantages and disadvantages of each scheme according to the preset engineering standards and objective functions, and finds the optimal pipeline laying scheme, thereby improving design efficiency and quality, and reducing labor costs and design risks.
[0005] This invention is achieved through the following technical solution: a method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion with A-satellite, comprising the following sub-steps: S1. Three-dimensional grid environment modeling of the marine platform: Based on the actual length, width and height of the marine platform compartments, the physical space is discretized into a three-dimensional grid matrix, and the start and end points of pipelines and inaccessible areas are marked. S2. Define the motion space that adapts to the three-dimensional linear movement characteristics of the pipeline of the offshore platform, construct and initialize the value evaluation matrix that associates the three-dimensional spatial coordinates with the motion, and initialize the relevant parameters for reinforcement learning. S3. Enter the training phase, adopting a round-based training mode, in which the agent explores the environment synchronously in each training round. S4. Based on the value evaluation matrix after training convergence, construct a heuristic function, design a dual cost evaluation system for the heuristic search algorithm, perform hybrid path search and reconstruction, and output the pipeline laying scheme.
[0006] In the three-dimensional grid matrix, each grid cell represents a spatial cell of fixed size. The size of the spatial cell is set according to the pipe diameter. Each grid cell is uniquely identified by three-dimensional coordinates (x, y, z). Each grid cell is configured with identification information to distinguish between free grids and obstacle grids. Free grids are identified by 0, indicating that the grid is a passable free space; obstacle grids are identified by 1, indicating that the grid is an impassable space with obstacles.
[0007] S2 includes the following sub-steps: S2-1. Considering the linear movement characteristics of pipeline laying in the three-dimensional space of the offshore platform, define an action space that adapts to the linear movement in the three-dimensional space. S2-2. Combine with S2-1 to construct a value assessment matrix, which is a four-dimensional Q-value matrix, and perform initialization settings for model learning parameters.
[0008] The action space in S2-1 contains 6 basic actions, each of which includes an action number, a reverse action number, an action direction number, and a spatial coordinate change. Among them, the action number is used to randomly select the action and action identifier, the reverse action number is used to identify the back action, the action direction number is used to determine the turn, and the spatial coordinate change is used to calculate the position of the next pipeline node in the grid space.
[0009] The value assessment matrix is represented as Q(x,y,z,a), where a is the action number, set to 0.1×rand, and rand is a random number between 0 and 1, used to store the value assessment of state (x,y,z)-action. The model learning parameters are initialized as follows: Set the initial greed factor ε_start = 0.95 and the final greed factor ε_end = 0.4; The decay rate ε_decay = 0.995, which decays dynamically in each training round; Learning rate =0.1, the learning rate decays dynamically, and the calculation formula is: _t= ×(1-episode / numEpisodes), in, _t represents the dynamically decaying learning rate, episode represents the current training epoch, and numEpisodes represents the total number of training epochs, to avoid oscillations in the later stages; Discount factor γ = 0.95; The number of agents is 2, the initial position of the agents is the pipeline start point, the experience pool capacity is 1000, and the random sampling batch size of the experience pool is 70.
[0010] S3 includes the following sub-steps: S3-1, Round Initialization: At the beginning of each training round, the state parameters are initialized. S3-2. Perform action selection. In the current state, select actions based on the greedy factor ε. The greedy factor gradually decreases as training progresses. S3-3. Perform state transition and validity check, execute the selected action, calculate the new position, and use a double verification mechanism to determine whether the executed action is valid. Calculate the reward value based on the judgment result and the multi-dimensional reward rules. S3-4. After the reward value is calculated, the value evaluation matrix is updated. The agent's effective actions, rewards and results are stored in the experience pool. When the number of experiences stored in the experience pool is greater than or equal to the random sampling batch, the Q value of the current state-action is updated according to the temporal difference learning rule. After completing the preset number of training rounds, the converged value evaluation matrix is obtained.
[0011] The initialization of state parameters in S3-1 is as follows: reset the current position to the pipeline starting point, the total reward to 0, the number of steps to 0, the arrival flag to not arrived, the previous action to no initial action, the identifier to -1, the turn count to 0, initialize the access mark matrix, and record the number of times the grid is accessed.
[0012] The dual verification mechanism in S3-3 includes determining whether the new position exceeds the boundary or belongs to the obstacle grid; If the new location exceeds the boundary and / or belongs to an obstacle grid, the executed action is deemed invalid and a penalty is imposed; If the new position does not exceed the boundary and is not within the obstacle grid, the executed action is deemed valid and a reward is given. The multi-dimensional reward rule calculation formula is as follows: ; in, ; ; ; ; ; ; In the formula, R is the total reward, r1 is the penalty for invalid actions, r2 is the end point reward, r3 is the end point movement reward and penalty, r4 is the exploration reward and penalty, r5 is the direct management reward, step is the number of action steps, and r6 is the agent collaborative exploration reward.
[0013] S4 includes the following sub-steps: S4-1. Construct a heuristic function based on the value evaluation matrix after training convergence, extract the optimal value evaluation value of all actions at each spatial location, and obtain the heuristic function hQL adapted to heuristic search after cost transformation, nonnegation and normalization. S4-2. Design a dual-cost evaluation system for heuristic search algorithms, where the dual costs include actual cost and movement cost; S4-3. Perform hybrid path search and reconstruction. Starting from the starting grid, traverse adjacent feasible grids and add them to the set to be explored. Select the grid with the lowest movement cost for expansion and record the parent node information. Recalculate the potential actual cost for the repeated grids and update the parent node and entry direction as needed. When the destination grid is reached, the optimized pipeline laying path is obtained by backtracking through the parent node information and output.
[0014] In S4-2, the actual cost record is the cumulative cost from the starting point to the current position, and its calculation formula is as follows: gScore = L + 2 * W; Where gScore is the actual cost, L is the distance cost of the path from the starting point to the current position, and W is the corner cost of the path from the starting point to the current position. The movement cost is obtained by weighted fusion of the actual cost and the heuristic cost, and the specific formula is as follows: fScore = gScore(CurrentPos) + 5× hQL(CurrentPos); Where fScore is the movement cost and CurrentPos is the current position.
[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention employs a technical approach combining 3D grid modeling, dual-agent reinforcement learning, and AlphaGo hybrid fine-tuning. Based on the Q-Learning algorithm, it designs a dedicated reward function, introduces a dual-agent collaborative training and experience replay mechanism, and integrates AlphaGo algorithm for path fine-tuning. This achieves multi-objective optimization of path feasibility, shortest path, and fewest turns. It fundamentally solves the problems of low efficiency, difficulty in turn optimization, and high collision risk in traditional manual design, as well as the poor constraint adaptability, low convergence efficiency, and insufficient adaptability to large-scale spaces of existing intelligent algorithms.
[0016] This application introduces a dual-agent collaborative training and experience replay mechanism. The two agents explore the environment synchronously from the starting point, and experience is shared by combining exclusive agent collaborative exploration rewards, which greatly improves the global exploration efficiency of the algorithm and avoids the problem of single agent frameworks being prone to getting trapped in local optima. The setting of the experience pool enables random sampling and batch updating of training data, making Q-table updates more efficient and stable. It solves the defects of traditional reinforcement learning algorithms, such as single training data and oscillations in the later convergence stage, and can maintain a high training success rate even in large-scale spaces.
[0017] This application creatively integrates Q-Learning reinforcement learning with the A* algorithm, transforming the converged Q-table into an A* algorithm heuristic function hQL. This allows the A* algorithm's heuristic function to incorporate the global experience of reinforcement learning, solving the problems of subjective heuristic function design and poor adaptability to complex constraints in the A* algorithm. Simultaneously, by leveraging the greedy search characteristics of the A* algorithm, it compensates for the shortcomings of pure reinforcement learning, such as insufficient path optimality and susceptibility to redundant movements. This fusion approach inherits Q-Learning's strong adaptability to complex obstacles and densely constrained scenarios while leveraging the A* algorithm's advantages in precise path search, thus overcoming the core technical bottlenecks of existing single-algorithm approaches.
[0018] The technical solution of this invention can be adapted to multi-scale marine platform spaces, breaking through the limitation of existing algorithms that can only adapt to small and medium-scale spaces, and is in line with the actual engineering scenarios of compact and large-scale layout of deep-water and ultra-deep-water marine platform compartments.
[0019] Compared to the basic Q-Learning algorithm and existing intelligent algorithms, the pipeline scheme generated by this invention is superior, with higher training efficiency and reduced repeated debugging during the design process. At the same time, the algorithm avoids problems such as pipeline collision, path redundancy, and excessive bends from the design source, reducing the engineering costs of pipeline construction and maintenance, as well as the safety risks of platform operation in the later stage. Attached Figure Description
[0020] Figure 1 This is a flowchart of the method used in this application; Figure 2 This is a schematic diagram of the layout result of this application in Environment 1 of Example 2; Figure 3 This is a schematic diagram of the layout results of the basic Q-Learning algorithm in Environment 1 of Example 2; Figure 4 This is a schematic diagram of the layout result of this application in Environment 2 of Example 2; Figure 5 This is a schematic diagram of the layout results of the basic Q-Learning algorithm in Environment 2 of Example 2; Figure 6 This is a schematic diagram of the layout result of this application in Environment 3 of Example 2; Figure 7 This is a schematic diagram of the layout results of the basic Q-Learning algorithm in Environment 3 of Example 2. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0022] Example 1 To better apply automated pipeline laying methods to offshore platform pipeline design, it is necessary to first improve the computational efficiency and convergence speed of the algorithm, and second, to align with actual engineering conditions, prioritizing orthogonalization feasibility while reducing engineering costs. Therefore, this application addresses the engineering requirements of offshore platform pipeline laying, which are characterized by limited space, dense obstacles, and strict control of engineering constraints. It proposes a pipeline laying method for offshore platforms based on dual-agent reinforcement learning fused with A-Star algorithm. This method employs a hybrid approach of 3D grid modeling, multi-agent reinforcement learning (Q-Learning algorithm), and A-Star algorithm fine-tuning. Building upon the Q-Learning algorithm's reward function design tailored to pipeline laying, it introduces a dual-agent collaborative training and experience playback mechanism, and integrates the A-Star algorithm for path fine-tuning. This achieves automated pipeline laying with multi-objective optimization of path feasibility, shortest path, and fewest turns, solving the problems of low efficiency, difficult turn optimization, and high collision risk associated with traditional manual design.
[0023] Reference Figure 1 The method proposed in this application includes the following steps: S1. Three-dimensional grid environment modeling of the offshore platform: Based on the actual length, width and height of the offshore platform compartments, the physical space is discretized into a three-dimensional grid matrix Grid (3×N, where N is the total number of grids), and the start and end points of pipelines and inaccessible areas (such as the space occupied by offshore platform equipment) are marked.
[0024] Each grid cell represents a spatial unit of fixed size. The size of the spatial unit is set according to the pipe diameter and is adapted to the pipe diameter. Each grid cell is uniquely identified by three-dimensional coordinates (x, y, z) and satisfies: 1≤x≤X_max, 1≤y≤Y_max, 1≤z≤Z_max; X_max, Y_max, and Z_max are the maximum number of grid cells in the corresponding axis of the marine platform compartment.
[0025] Each grid cell is configured with identification information to distinguish between free grids and obstacle grids. Free grids are identified by 0, indicating that the grid is a passable free space; obstacle grids are identified by 1, indicating that the grid is an impassable space with obstacles.
[0026] S2. Define the motion space that adapts to the three-dimensional linear movement characteristics of the pipeline on the offshore platform, construct and initialize the value evaluation matrix that associates the three-dimensional spatial coordinates with the motion, and initialize the relevant parameters for reinforcement learning.
[0027] S2 includes the following sub-steps: S2-1. Considering the linear movement characteristics of pipeline laying in the three-dimensional space of the offshore platform, define an action space adapted to the linear movement in the three-dimensional space. The action space contains 6 basic actions, each of which includes an action number, a reverse action number, an action direction number, and a change in spatial coordinates.
[0028] Furthermore, the relevant parameters for these six basic movements are shown in Table 1.
[0029] Table 16 Parameters Related to Basic Movements Action Number Reverse action number Action direction number Spatial coordinate transformation 1 2 0 (positive x-axis direction) (1, 0, 0) 2 1 0 (negative x-axis direction) (-1, 0, 0) 3 4 1 (positive y-axis direction) (0, 1, 0) 4 3 1 (negative y-axis direction) (0, -1, 0) 5 6 2 (positive z-axis direction) (0, 0, 1) 6 5 2 (negative z-axis direction) (0, 0, -1) Among them, the action number is used to randomly select the action and action identifier, the reverse action number is used to identify the back action, the action direction number is used to determine the turn, and the spatial coordinate change is used to calculate the position of the next pipeline node in the grid space.
[0030] S2-2. Combine with S2-1 to construct a value assessment matrix, which is a four-dimensional Q-value matrix, and perform model learning parameter initialization settings. The Q-value matrix is represented as Q(x,y,z,a), where a is the action number. Its initial value is set to 0.1×rand to avoid initial bias by using a random small value, and rand is a random number between 0 and 1, used to store the value evaluation of state (x,y,z)-action.
[0031] The learning parameters for the Q-Learning algorithm are configured as follows: Set the initial greed factor ε_start=0.95 (focusing on exploration in the early stages) and the final greed factor ε_end=0.4 (focusing on utilizing the optimal path in the later stages). The decay rate ε_decay = 0.995, which decays dynamically in each training round; Learning rate =0.1, the learning rate decays dynamically, and the calculation formula is: _t= ×(1-episode / numEpisodes), in, _t represents the dynamically decaying learning rate, episode represents the current training epoch, and numEpisodes represents the total number of training epochs, to avoid oscillations in the later stages; With a discount factor γ=0.95, it emphasizes future rewards and guides the long-term optimal path; Set the maximum number of steps per round (maxSteps) and the total number of training rounds (numEpisodes).
[0032] The number of agents is set to 2, the initial position of the agents is the pipeline start point, the experience pool capacity is 1000, and the random sampling batch size of the experience pool is 70.
[0033] S3. Enter the training phase, adopting a turn-based training mode, in which the agent explores the environment synchronously in each training round.
[0034] Specifically, S3 includes the following steps: S3-1, Round Initialization: At the beginning of each training round, the state parameters are initialized. Specifically, at the start of each training episode, the current position CurrentPos is reset to the pipeline start point StartPos, which is represented as CurrentPos=StartPos; the total reward is 0, which is represented as TotalReward=0; the number of steps is 0, which is represented as Steps=0; the reached goal is not reached, which is represented as Reached Goal=false; the previous action is no initial action, marked as -1, which is represented as Prev Action=-1; the turn count is 0, which is represented as Turn Count=0; at the same time, the visit mark matrix Visited is initialized to record the number of times the grid is visited to avoid repeated exploration. 0 indicates that it has not been visited, and non-zero numbers indicate the number of visits.
[0035] S3-2. Perform action selection. In the current state, select actions based on the greedy factor ε. The greedy factor gradually decreases as training progresses. The greedy factor ε = max(ε_end, ε_start × ε_decay).
[0036] Set a random number. If the random number is less than ε, randomly select an action to avoid local optima. If the random number is greater than ε, select the action with the largest Q value in the current state.
[0037] S3-3. Perform state transition and validity check, execute the selected action, calculate the new position NextPos=CurrentPos+action, and use a double verification mechanism to determine whether the executed action is valid. Calculate the reward value based on the judgment result and the multi-dimensional reward rules.
[0038] The dual verification mechanism includes determining whether the new position NextPos exceeds the boundary and whether it belongs to an obstacle grid.
[0039] If the new location exceeds the boundary and / or belongs to an obstacle grid, the executed action is deemed invalid and a penalty is imposed; If the new position does not exceed the boundary and is not within the obstacle grid, the executed action is deemed valid and a reward is given.
[0040] After the action is completed, a reward is calculated. This embodiment uses a multi-dimensional reward function to maximize the achievement of three objectives: path feasibility, movement towards the target, and reduction of turns. The expression for the multi-dimensional reward function is: ; in, ; ; ; ; ; ; In the formula, R is the total reward, r1 is the penalty for invalid actions, r2 is the end point reward, r3 is the end point movement reward and penalty, r4 is the exploration reward and penalty, r5 is the direct management reward, step is the number of action steps, and r6 is the agent collaborative exploration reward.
[0041] Please refer to Table 2 for specific reward rules.
[0042] Table 2. Detailed List of Reward Rules Action execution result Reward Value Design Goals Crossing the boundary / collision with obstacles -50 Avoid collisions between pipelines and platform structure Reaching the end of the pipeline 1000-step The reward increases as the number of steps decreases, guiding the path to the shortest pipeline. Move towards the end of the pipeline +3 Avoid ineffective detours Far from the end of the pipeline -1 Reduce redundancy length Access new grid +1 Encourage exploration of uncovered areas to avoid local optima. Repeatedly access the grid (>5 times) -2 Penalize loop paths to improve training success rate Bend -10 Reducing the number of pipe bends lowers fluid resistance and construction difficulty. straight pipe +2 Incentivize straight pipes and optimize pipeline laying processes. The agent explores the same area +1 Promote the sharing of experiences and improve exploration efficiency S3-4. After the reward value is calculated, the value evaluation matrix (Q-value matrix) is updated. The agent's effective actions, rewards and results are stored in the experience pool. When the number of experiences stored in the experience pool is greater than or equal to the random sampling batch, the Q-value of the current state-action is updated according to the temporal difference learning rule. After completing the preset number of training rounds, the converged state-action value evaluation matrix is obtained.
[0043] The formula for updating the Q value is: Q(CurrentPos, action) = _t×[Reward+γ×max(Q(NextPos,:))-Q(CurrentPos, action)]+Q(CurrentPos, action); In the formula, max(Q(NextPos,:)) is the optimal Q value of the new position after the action is performed, and Reward is the reward value after the action is performed; "=" is the assignment operation, where Q(CurrentPos, action) on the left side of "=" is the updated Q value, and Q(CurrentPos, action) on the right side is the Q value before the update.
[0044] When the experience pool reaches its maximum capacity, the oldest experience points are deleted.
[0045] S4. Based on the value evaluation matrix after training convergence, construct a heuristic function, design a dual cost evaluation system for the heuristic search algorithm, perform hybrid path search and reconstruction, and output the pipeline laying scheme.
[0046] To improve the completeness of the solution computation based on the converged Q-table after training, the Q-Learning algorithm and the A* algorithm are combined and fine-tuned to ensure that the path reaches the optimal value as much as possible.
[0047] S4 includes the following sub-steps: S4-1. Constructing a heuristic function based on the value evaluation matrix after training convergence is the core connecting the Q-Learning algorithm and the A* algorithm. It extracts the optimal value evaluation value of all actions at each spatial location, and after cost transformation, non-negativity, and normalization, obtains a heuristic function adapted to heuristic search. Specifically, for each spatial location in the 3D grid, the maximum Q-value of all possible actions in that state is extracted. The Q-value contains the comprehensive benefit (including path length, turning cost, etc.) from that location to the destination learned by the Q-Learning algorithm through extensive training. To adapt to the search logic of the A* algorithm, the maximum Q-value is transformed: first, it is negativeized so that high-benefit Q-values correspond to low heuristic costs, then shifted to a non-negative interval and normalized to the [0,1] range, forming the final heuristic function hQL. This ensures that during the hybrid fine-tuning process, the considered better path direction will obtain a lower heuristic cost.
[0048] S4-2. Design a dual-cost evaluation system for heuristic search algorithms, where the dual costs include actual cost and movement cost.
[0049] Based on S4-2, this application constructs the core mechanism of the A* algorithm to adapt to pipeline laying requirements. The actual cost, gScore, records the cumulative cost from the starting point to the current position. This includes not only the basic movement cost (each grid movement is counted as 1, directly related to the pipeline length) but also a bend penalty (when the current movement direction differs from the previous direction, an additional 2 units of cost are added, consistent with the turning penalty logic of Q-Learning, reinforcing the goal of "fewer turns"). The specific formula is as follows: gScore = L + 2 * W; In the formula, L is the distance cost of the path from the starting point to the current position, and W is the corner cost of the path from the starting point to the current position.
[0050] The movement cost fScore is obtained by weighted fusion of the actual cost and the heuristic cost, and the specific formula is as follows: fScore = gScore(CurrentPos) + 5 × hQL(CurrentPos).
[0051] This allows the search process to consider both the actual costs incurred and the full utilization of the global experience of Q-Learning.
[0052] S4-3. Perform hybrid path search and reconstruction. Starting from the starting grid, traverse adjacent feasible grids and add them to the set to be explored. Select the grid with the lowest movement cost for expansion and record the parent node information. Recalculate the potential actual cost for the repeated grids and update the parent node and entry direction as needed. When the destination grid is reached, the optimized pipeline laying path is obtained by backtracking through the parent node information and output.
[0053] Furthermore, this application starts from the starting grid, traverses the adjacent feasible grids of the current grid and adds them to the openSet to be explored. The grid with the lowest total cost in the openSet is continuously selected for expansion, while the parent node is recorded. If a duplicate grid is encountered, its potential actual cost is recalculated. If the actual cost is lower than the historical record, the parent node and the entry direction are updated. When the search reaches the endpoint grid, the entire path is backtracked through the parent node record. This path inherits the adaptability of Q-Learning to complex obstacle environments and shortens the path length through the greedy search characteristics of A-Learning, ultimately forming an optimal pipeline laying scheme for offshore platforms that balances fewer turns, shorter paths, and no collisions. The specific process is as follows: Initial settings: CurrentPos=StartPos, openSet=[StartPos], Path Turn Count=0, Prev Action=-1.
[0054] Path search: Calculate the movement cost fScore of the adjacent feasible positions of the current position CurrentPos, add it to openSet, select the position with the minimum movement cost in openSet to move to, record the movement path, and continuously iterate the search until openSet is empty or the end position of the pipeline is reached. Then, the backtracking path is recorded through the parent node to form the final pipeline laying scheme and output it.
[0055] This application obtains the coordinates of the pipeline nodes in three-dimensional space by backtracking the parent node records, and reconstructs the solution vector into a pipeline node coordinate matrix for visualization output.
[0056] Example 2 To demonstrate the effectiveness of the method described in Example 1, such as Figures 2 to 7 As shown, this embodiment conducts experimental simulations in grid map environments with different obstacles and different start and end points (Environment 1: 20×20×20 grid, Environment 2: 30×30×30 grid, Environment 3: 35×35×35 grid). Figures 2 to 7 In the diagram, green dots represent the starting point of the pipeline, blue dots represent the ending point of the pipeline, and black lines represent the pipeline laying path.
[0057] This embodiment is compiled using MATLAB in a Windows 11 environment. First, a simulation model space for obstacles is established. The device models in the layout space can be regarded as obstacles that cannot be laid with pipelines. The grid method is used to divide the space equally, with a grid granularity of 1. The total number of training rounds numEpisodes and the maximum number of steps per round maxSteps are both 500.
[0058] This application is compared with the basic Q-Learning algorithm, and the comparison data is shown in Table 3.
[0059] Table 3 Comparison between this application and the basic Q-Learning algorithm
[0060] Experimental results show that the algorithm proposed in this invention has good performance in pipeline layout in different marine platform environments. It significantly reduces the number of bends (e.g., from 46 to 2 in a 35×35×35 environment) and further optimizes the path length while reducing the number of bends (e.g., from 75 to 73 in a 35×35×35 environment), effectively reducing pipeline fluid resistance and construction difficulty. The entire path generation process is checked for boundaries and obstacles to ensure no collisions, improving the safety and manufacturability of pipeline laying and truly achieving global optimization for multiple objectives.
[0061] This application maintains a high training success rate of 93.7% even in large-scale spaces (such as 35×35×35 environments), exhibiting excellent path optimization and stability, and significantly improving the engineering adaptability of intelligent design algorithms.
[0062] In summary, this invention, through architectural innovation of reinforcement learning algorithms and deep integration with the A* algorithm, improves the standard algorithm into a dedicated method for efficiently solving the three-dimensional pipeline laying problem of marine platforms. This significantly enhances the design quality, search efficiency, and engineering adaptability of pipeline laying for marine platforms, providing a brand-new technical solution for the intelligent design of pipelines for deep-water and ultra-deep-water marine platforms.
[0063] The above description is merely an optional embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the content of the present invention under the concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. A method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion with satellite A, characterized in that, Includes the following sub-steps: S1. Three-dimensional grid environment modeling of the marine platform: Based on the actual length, width and height of the marine platform compartments, the physical space is discretized into a three-dimensional grid matrix, and the start and end points of pipelines and inaccessible areas are marked. S2. Define the motion space that adapts to the three-dimensional linear movement characteristics of the pipeline of the offshore platform, construct and initialize the value evaluation matrix that associates the three-dimensional spatial coordinates with the motion, and initialize the relevant parameters for reinforcement learning. S3. Enter the training phase, adopting a round-based training mode, in which the agent explores the environment synchronously in each training round. S4. Based on the value evaluation matrix after training convergence, construct a heuristic function, design a dual cost evaluation system for the heuristic search algorithm, perform hybrid path search and reconstruction, and output the pipeline laying scheme.
2. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 1, characterized in that, In the three-dimensional grid matrix, each grid cell represents a spatial cell of fixed size. The size of the spatial cell is set according to the pipe diameter. Each grid cell is uniquely identified by three-dimensional coordinates (x, y, z). Each grid cell is configured with identification information to distinguish between free grids and obstacle grids. Free grids are identified by 0, indicating that the grid is a passable free space; obstacle grids are identified by 1, indicating that the grid is an impassable space with obstacles.
3. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 1, characterized in that, S2 includes the following sub-steps: S2-1. Considering the linear movement characteristics of pipeline laying in the three-dimensional space of the offshore platform, define an action space that adapts to the linear movement in the three-dimensional space. S2-2. Combine with S2-1 to construct a value assessment matrix, which is a four-dimensional Q-value matrix, and perform initialization settings for model learning parameters.
4. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 3, characterized in that, The action space in S2-1 contains 6 basic actions, each of which includes an action number, a reverse action number, an action direction number, and a spatial coordinate change. Among them, the action number is used to randomly select the action and action identifier, the reverse action number is used to identify the back action, the action direction number is used to determine the turn, and the spatial coordinate change is used to calculate the position of the next pipeline node in the grid space.
5. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 3, characterized in that, The value assessment matrix is represented as Q(x,y,z,a), where a is the action number, set to 0.1×rand, and rand is a random number between 0 and 1, used to store the value assessment of state (x,y,z)-action. The model learning parameters are initialized as follows: Set the initial greed factor ε_start = 0.95 and the final greed factor ε_end = 0.4; The decay rate ε_decay = 0.995, which decays dynamically in each training round; Learning rate =0.1, the learning rate decays dynamically, and the calculation formula is: _t= ×(1-episode / numEpisodes), in, _t represents the dynamically decaying learning rate, episode represents the current training epoch, and numEpisodes represents the total number of training epochs, to avoid oscillations in the later stages; Discount factor γ = 0.95; The number of agents is 2, the initial position of the agents is the pipeline start point, the experience pool capacity is 1000, and the random sampling batch size of the experience pool is 70.
6. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 3, characterized in that, S3 includes the following sub-steps: S3-1, Round Initialization: At the beginning of each training round, the state parameters are initialized. S3-2. Perform action selection. In the current state, select actions based on the greedy factor ε. The greedy factor gradually decreases as training progresses. S3-3. Perform state transition and validity check, execute the selected action, calculate the new position, and use a double verification mechanism to determine whether the executed action is valid. Calculate the reward value based on the judgment result and the multi-dimensional reward rules. S3-4. After the reward value is calculated, the value evaluation matrix is updated. The agent's effective actions, rewards and results are stored in the experience pool. When the number of experiences stored in the experience pool is greater than or equal to the random sampling batch, the Q value of the current state-action is updated according to the temporal difference learning rule. After completing the preset number of training rounds, the converged value evaluation matrix is obtained.
7. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 6, characterized in that, The initialization of state parameters in S3-1 is as follows: reset the current position to the pipeline starting point, the total reward to 0, the number of steps to 0, the arrival flag to not arrived, the previous action to no initial action, the identifier to -1, the turn count to 0, initialize the access mark matrix, and record the number of times the grid is accessed.
8. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 6, characterized in that, The dual verification mechanism in S3-3 includes determining whether the new position exceeds the boundary or belongs to the obstacle grid; If the new location exceeds the boundary and / or belongs to an obstacle grid, the executed action is deemed invalid and a penalty is imposed; If the new position does not exceed the boundary and is not within the obstacle grid, the executed action is deemed valid and a reward is given. The multi-dimensional reward rule calculation formula is as follows: ； in, ; ； ； ； ； ； In the formula, R is the total reward, r1 is the penalty for invalid actions, r2 is the end point reward, r3 is the end point movement reward and penalty, r4 is the exploration reward and penalty, r5 is the direct management reward, step is the number of action steps, and r6 is the agent collaborative exploration reward.
9. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 1, characterized in that, S4 includes the following sub-steps: S4-1. Construct a heuristic function based on the value evaluation matrix after training convergence, extract the optimal value evaluation value of all actions at each spatial location, and obtain the heuristic function hQL adapted to heuristic search after cost transformation, nonnegation and normalization. S4-2. Design a dual-cost evaluation system for heuristic search algorithms, where the dual costs include actual cost and movement cost; S4-3. Perform hybrid path search and reconstruction. Starting from the starting grid, traverse adjacent feasible grids and add them to the set to be explored. Select the grid with the lowest movement cost for expansion and record the parent node information. Recalculate the potential actual cost for the repeated grids and update the parent node and entry direction as needed. When the destination grid is reached, the optimized pipeline laying path is obtained by backtracking through the parent node information and output.
10. The method for laying pipelines on marine platforms based on dual-agent reinforcement learning fusion A-satellite as described in claim 9, characterized in that, In S4-2, the actual cost record is the cumulative cost from the starting point to the current position, and its calculation formula is as follows: gScore = L + 2 * W; Where gScore is the actual cost, L is the distance cost of the path from the starting point to the current position, and W is the corner cost of the path from the starting point to the current position. The movement cost is obtained by weighted fusion of the actual cost and the heuristic cost, with the specific formula as follows: fScore = gScore(CurrentPos) + 5× hQL(CurrentPos); Where fScore is the movement cost and CurrentPos is the current position.