A warship power system equipment task simulation method and system
By creating a power mission sequence and information pool, and combining the Q-learning algorithm and Markov decision process, the problems of lag and low matching of ship power equipment mission models were solved, achieving real-time and accurate simulation results, and improving simulation efficiency and matching with actual combat missions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA STATE SHIPBUILDING CORP LTD RESEARCH INSTITUTE 719
- Filing Date
- 2023-03-17
- Publication Date
- 2026-08-04
AI Technical Summary
Existing ship propulsion equipment mission models suffer from lag and low matching, resulting in models that can only provide reference for real-time propulsion equipment mission execution sequences, rather than providing accurate and effective real-time guidance. Furthermore, performance analysis is prone to dogmatism and template bias.
A simulation method for ship propulsion system equipment is adopted. By creating a propulsion mission sequence, establishing a first environmental information pool and a second action information pool, and utilizing the Q-learning algorithm and Markov decision process, combined with a fixed-step long-time advancement mechanism, simulation time points are generated in real time, the propulsion equipment mission sequence is optimized, and the matching logic relationship and simulation efficiency are improved.
This method achieves real-time performance and accuracy in naval propulsion system equipment mission simulation, improves simulation efficiency, reduces time costs, and more closely reflects actual combat missions, providing a more accurate basic module for naval propulsion system mission simulation.
Smart Images
Figure CN116451562B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of ship RMS simulation modeling equipment mission simulation, and more specifically, relates to a ship power system equipment mission simulation method and system. Background Technology
[0002] Naval equipment is a multi-level, complex system that includes both electromechanical and electronic equipment that requires long-term operation and auxiliary combat equipment that operates intermittently on watch. As the core equipment system supporting ship-related missions, the ship propulsion system requires precise RMS modeling and simulation. This accurate and effective analysis of the temporal relationships of various typical mission profiles within a specific mission sequence is crucial for fully utilizing limited resources and meeting the ship's combat effectiveness and endurance requirements.
[0003] Existing methods for establishing mission models for ship propulsion equipment primarily utilize traditional empirical and mathematical analytical methods. They employ Vensim software to construct corresponding stock flow diagrams, use fixed growth curve models, and combine subjective expert scoring with comprehensive evaluation to analyze and assess the model's modeling and simulation performance. The drawback of this approach is that the model parameters and data used for modeling are not dynamically changing but rather rely on previously recorded experimental data. This results in inherent lag and low model matching, meaning the model only provides reference value for real-time propulsion equipment mission sequences and cannot provide accurate and effective real-time guidance. Furthermore, the model analysis and evaluation employ subjective methods, entirely dependent on the skill level and experience of the evaluation experts. This leads to significant biases in the model's effectiveness analysis due to the rigidity and template-based nature of empirical dogma. Therefore, computer modeling and simulation, as an advanced technology, can more accurately and effectively solve complex system simulation problems. Computers use established simulation models of actual systems to simulate the system's operating state and its temporal variations. By observing and statistically analyzing the simulation process, the simulation output parameters and basic characteristics of the simulated system are obtained, thereby estimating and inferring the actual parameters and performance of the real system. It is a simulation technology with good controllability, non-destructive nature, and high efficiency and economy. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for simulating ship propulsion system equipment tasks. This method and system address the problems of existing ship propulsion equipment task models, which suffer from inherent lag and low matching, resulting in models that only provide reference for real-time power equipment task execution sequences and cannot provide accurate and effective real-time guidance. Furthermore, the effectiveness analysis of the models is subject to serious biases due to the rigidity and template-based nature of experience and dogma.
[0005] To achieve the above objectives, the present invention provides a method for simulating the mission of a ship's propulsion system, comprising the following steps:
[0006] D1: Create a power mission sequence and supplement the power system mission sequence with the entity information of the power equipment in the combat method; wherein, the power system mission sequence is a mission sequence generated by randomly sorting the missions executed in the ship's power equipment;
[0007] D2: Based on the entity information of the power equipment in the combat method, establish the first environmental information pool; wherein, the first environmental information pool is composed of random variables generated at the moment of the occurrence of random events; the random event is the status of a certain mechanical part in the power system parts library during the execution of the mission;
[0008] D3: Load the entity information of each power equipment and the simulated combat mission of each power equipment for each environmental information item in the first environmental information pool, and form the equipment information and simulated combat mission into equipment queue and event queue respectively.
[0009] D4: Establish a second action information pool using an event queue;
[0010] D5: Assign the simulated combat missions in the event queue to the various second action information in the second action information pool to obtain the execution action queue of each second action information pool;
[0011] D6: The simulation process is advanced by a fixed step time advance mechanism. The equipment queue is used to build a reward function to determine the current system simulation clock size. Simulated combat missions only exist within the fixed step time, and there are no simulated combat missions during the time interval.
[0012] D7: Based on the second action information in the second action information pool, perform simulation on each simulated combat task in the event queue corresponding to each second action information according to the current system simulation clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed.
[0013] More preferably, the method for obtaining the random variable includes the following steps:
[0014] Generate a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator;
[0015] The random number sequence is normalized using a modulus.
[0016] Random variables are obtained by using the inverse transformation method and function transformation method of the normalized random number sequence.
[0017] More preferably, the method for establishing the second action information pool using simulated combat mission information is as follows:
[0018] Clustering tree diagram analysis is performed on the structural hierarchy of simulated combat mission information to establish mission logic. Each action information in the second action information pool is digitally described by the mission profile and the mission stage corresponding to each mission profile.
[0019] The information for each action is as follows: 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm. To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile includes air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile; the number of second action information items in the second action information pool is...
[0020] Where, N J The number of task profiles J; The number of task phases;
[0021] The second action information pool contains four action sets: air combat set, anti-ship attack set, anti-submarine attack set, and land attack set.
[0022] More preferably, D7 specifically includes the following steps:
[0023] D7.1: Set the initial events in the dynamic system task sequence to random states;
[0024] D7.2: Based on the task sequence of the power system, determine whether the current system simulation time is within a fixed step time. If so, obtain the current system simulation clock and proceed to D7.3; otherwise, obtain the simulation combat task corresponding to the current system simulation clock from the execution action queue corresponding to each second action information and proceed to D7.4.
[0025] D7.3: Set the end time of the fixed step time corresponding to the current system simulation time to the current system simulation time, then go to D7.2;
[0026] D7.4: Determine whether the current system simulation time is less than the end time of the current simulated combat mission based on the information of each second action. If so, proceed to D7.5; otherwise, move the current simulated combat mission to the historical event queue and proceed to D7.6.
[0027] D7.5: The second action information is executed and interacted with different environmental information in the first environmental information pool to obtain sample sequence and reward sequence. The state-action value function in the Q-learning algorithm is introduced to perform simulation iterative operation, and the simulation process and simulation results are cached.
[0028] D7.6: Determine whether the corresponding execution action has been executed for each second action information. If yes, proceed to D7.7; otherwise, the remaining execution action sequence for each second action information is executed continuously within a fixed step size, and return to D7.5.
[0029] D7.7: Determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, output the cached result; otherwise, adjust the fixed step time and proceed to step 7.2.
[0030] On the other hand, the present invention provides a ship propulsion system equipment mission simulation system, comprising:
[0031] The power mission sequence construction module is used to create power mission sequences and supplement the power system mission sequence with the entity information of the power equipment in the combat method; wherein, the power system mission sequence is a mission sequence generated by randomly sorting the missions executed in the ship's power equipment.
[0032] The first environmental information pool construction module is used to establish a first environmental information pool based on the entity information of the power equipment in the combat method; wherein, the first environmental information pool is composed of random variables generated at the time of the occurrence of random events; the random event is the status of a certain mechanical part in the power system parts library during the execution of a mission;
[0033] The equipment-event queue construction module is used to load the entity information of each power equipment and the simulated combat mission of each power equipment for each piece of environmental information in the first environmental information pool, and to form equipment queues and event queues for each piece of equipment information and simulated combat missions respectively.
[0034] The second action information pool construction module is used to establish a second action information pool using an event queue.
[0035] The action queue construction module is used to allocate simulated combat tasks in the event queue to various second action information in the second action information pool, thereby obtaining the action queues of each second action information pool.
[0036] The current system simulation clock determination module is used to advance the simulation process using a fixed step time advance mechanism as the simulation clock. It uses an equipment queue to build a reward function to determine the size of the current system simulation clock. Simulated combat missions only exist within the fixed step time, and there are no simulated combat missions during the time interval.
[0037] The simulation execution module is used to simulate each simulated combat task in the event queue corresponding to each second action information based on the second action information pool and according to the current system simulation clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed.
[0038] More preferably, the method for obtaining the random variable includes the following steps:
[0039] Generate a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator;
[0040] The random number sequence is normalized using a modulus.
[0041] Random variables are obtained by using the inverse transformation method and function transformation method of the normalized random number sequence.
[0042] More preferably, the method by which the second action information pool construction module establishes the second action information pool is as follows:
[0043] Clustering tree diagram analysis is performed on the structural hierarchy of simulated combat mission information to establish mission logic. Each action information in the second action information pool is digitally described by the mission profile and the mission stage corresponding to each mission profile.
[0044] The information for each action is as follows: 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm. To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile includes air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile; the number of second action information items in the second action information pool is...
[0045] Where, N J The number of task profiles J; The number of task phases;
[0046] The second action information pool contains four action sets: air combat set, anti-ship attack set, anti-submarine attack set, and land attack set.
[0047] More preferably, the simulation execution module includes:
[0048] An initial event setting unit is used to set the initial events in the power system task sequence to a random state.
[0049] The current system simulation time determination unit is used to determine whether the current system simulation time is within a fixed step time based on the power system task sequence. If so, it obtains the current system simulation clock and sets the end time of the fixed step time corresponding to the current system simulation time as the current system simulation time. It also determines whether the current system simulation time is less than the end time of the current simulated combat mission based on each second action information. If not, it moves the current simulated combat mission to the historical event queue.
[0050] The simulated combat mission acquisition unit is used to acquire the simulated combat mission corresponding to the current system simulation clock from the execution action queue corresponding to each second action information.
[0051] The simulation unit is used to execute the action of each second action information and interact with different environmental information in the first environmental information pool to obtain sample sequence and reward sequence. The state-action value function in the Q-learning algorithm is introduced to perform simulation iterative operation, and the simulation process and simulation results are cached.
[0052] The execution determination unit for each action is used to determine whether the corresponding action has been executed based on each second action information. If not, the remaining action sequence corresponding to each second action information is executed continuously within a fixed step size, driving the simulation unit to execute.
[0053] The execution determination unit of the second action information pool is used to determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, the cached result is output; otherwise, the fixed step time is adjusted to drive the determination unit of the current system simulation time to run.
[0054] In summary, compared with the prior art, the above-described technical solutions conceived by this invention have the following advantages:
[0055] Beneficial effects:
[0056] This invention provides a method and system for simulating ship propulsion system equipment missions. This invention loads each piece of first environmental information in the first environmental information pool constructed by the propulsion system mission sequence onto each propulsion equipment information and simulated combat mission, builds the state space in the Markov decision model, and improves the matching logic relationship between various propulsion equipment and combat missions.
[0057] This invention provides a method and system for simulating ship propulsion system equipment missions. The method generates simulation time points in real time by using simulation parameters related to the propulsion system mission sequence and a fixed-step simulation clock advancement mechanism. The simulation process is advanced by allocating simulated combat missions in the event queue to various second action information in the second action information pool through the propulsion system mission sequence, thus obtaining an executable action queue for each second action information and establishing an action space in the Markov decision model. Each second action information is selected sequentially by the agent from each executable action sequence based on the current system simulation time, and continuously interacts with environmental information. Guided by the Markov decision model's reward function, this process continues until each action information's executable action queue has been selected and executed once. This yields a change in equipment information state and an immediate reward after each combat mission is selected and executed, thereby updating and optimizing the propulsion equipment mission sequence in real time. By advancing the simulation clock, the relative start interval and allowable delay time of combat missions between mission profiles in the mission sequence are obtained. Based on this, the relationship between missions and equipment is established, resulting in a complete simulation method for power system equipment missions. This method effectively saves time costs, improves the efficiency of large-scale discrete combat mission simulation for ships, and because reinforcement learning requires fewer constraints to define the model, it enables flexible loading of power equipment information and simulated combat missions, which is closer to the actual combat mission situation of ships. This results in a more realistic and accurate simulation method for ship power system missions, providing a callable basic module for overall ship RMS simulation modeling. Attached Figure Description
[0058] Figure 1 This is a flowchart of the method for establishing a mission model of a ship propulsion system provided in an embodiment of the present invention;
[0059] Figure 2 This is a task hierarchy description diagram provided in an embodiment of the present invention. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0061] like Figure 1 As shown, the present invention provides a method for simulating ship propulsion system equipment missions, comprising the following steps:
[0062] Step 1: Create a power system mission sequence and add the entity information of the power equipment in the combat method to the power system mission sequence;
[0063] Among them, the power system task sequence is a summary of the tasks involved in the ship's power equipment, and all the tasks are randomly sorted to generate a task sequence.
[0064] Step 2: Based on the entity information of the power equipment in the combat method, establish the first environmental information pool; more specifically:
[0065] The first environmental information pool, established based on the physical information of the power equipment, is formed by randomly sampling various random events generated by each piece of equipment during the equipment's lifespan, including failures during peacetime training or wartime missions. The random variables generated at the time of each random event constitute the first environmental information pool. More specifically, a random event is described as a mechanical part in the power system parts library being in a normal, worn, deformed, or broken state. Random variables are described as the time when various random events occur. For example, at 1030 seconds, the power system-propulsion device-main engine-connecting rod deforms and the piping system-cooling piping system-temperature control valve breaks; at 5800 seconds, the power system-auxiliary device-ship power station-distribution board wears and the transmission device-clutch wears and the drive shaft-intermediate shaft breaks.
[0066] More preferably, the method for obtaining the random variable is as follows:
[0067] The formula for generating a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator is as follows:
[0068] X n+1 =(aX n +c)(modm)n≥0
[0069] Where m is the modulus, a is the multiplier, c is the increment, the initial value X0 is the seed, and m > 0, m > a, m > c, X0 < m; the calculation is performed using the congruence multiplication method; the random number sequence is represented as (0.3, 0.6, 0.8, 0.2...), and the length of the sequence is determined by the modulus m;
[0070] Normalize the integer random number sequence using the modulus:
[0071]
[0072] Obtain a random number R in the interval [0,1]. n (n = 1, 2, ...), where 0 ≤ X n ≤(m-1);
[0073] Random variables are obtained by using inverse transformation and function transformation methods with uniformly distributed random numbers in the interval [0,1].
[0074] Step 3: Load the entity information of each power equipment and the simulated combat mission involving that equipment into each piece of environmental information in the first environmental information pool, and construct the entity information and simulated combat mission of each piece of equipment into an equipment queue and an event queue respectively.
[0075] Step 4: Establish a second action information pool using simulated combat mission information; the second action information pool contains four action sets, namely air combat set, anti-ship attack set, anti-submarine attack set, and land attack set. Among them, the anti-ship attack set contains four types of actions: transit and rendezvous, tactical deployment, attack, and withdrawal.
[0076] More specifically, step 4 includes the following steps:
[0077] like Figure 2 As shown, a clustering dendrogram analysis is performed on the hierarchical structure of simulated combat mission information to establish mission logic, consisting of mission profile J and the mission phase J corresponding to each item J. K Each action information item in the second action information pool is digitally described, and its formula is expressed as follows:
[0078]
[0079] Where 'a' represents each action information and also represents the action to be executed in the Q-learning algorithm based on the action space in Markov Decision Process; To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile is divided into air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile;
[0080] Depend on Figure 2 The formula shown To explain, task profile C J The task profile has four phases, C = 4. JK ; information about a certain action can be obtained from a formula Make a selection;
[0081] The number of second action information items in the second action information pool is:
[0082] Where, N J The number of task profiles J; The number of task phases;
[0083] Step 5: Assign the simulated combat tasks in the event queue to each second action information in the second action information pool to obtain the execution action queue of each second action information pool;
[0084] Step 6: The simulation process is advanced using a fixed step time advance mechanism as the simulation clock. A reward function is built using an equipment queue to determine the current system simulation global clock TIME size. The reward function is the reward size obtained by the power system task sequence when the environmental information in the first environmental information pool changes after the power system task sequence executes various actions in the second action information pool. Among them, the simulated combat task only exists within the fixed step time, and there is no simulated combat task during its time interval.
[0085] The reward function formula is as follows:
[0086]
[0087] Where, p t Represents time priority; p m Represents task priority, and p m The value range is [0.2, 0.4, 0.6, 0.8]. k is the weight parameter; λ is the time required for the event to be executed; α(t) is the action chosen by the agent at time t; and s(t) is the environmental state at time t.
[0088] Step 7: Based on the second action information in the second action information pool, perform simulation on each simulated combat task in the event queue corresponding to each second action information according to the current system simulation clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed; specifically including the following steps:
[0089] Step 7.1: Set the initial events in the dynamic system task sequence to random states;
[0090] Step 7.2: Based on the task sequence of the dynamic system, determine whether the current simulation time is within a fixed step time. If yes, obtain the current system simulation clock and proceed to step 7.3; otherwise, obtain the simulation combat task corresponding to the current system simulation clock from the execution action queue corresponding to each second action information and proceed to step 7.4.
[0091] Step 7.3: Set the end time of the fixed step period corresponding to the current system simulation time to the current system simulation time, and return to step 7.2;
[0092] Step 7.4: Determine whether the current system simulation time is less than the end time of the current simulated combat mission based on each second action information. If so, proceed to step 7.5; otherwise, move the current simulated combat mission to the historical event queue and proceed to step 7.6.
[0093] Step 7.5: Interact each second action information with different environmental information in the first environmental information pool to obtain a large number of sample sequences. The corresponding return sequence is The state-action value function from the Q-learning algorithm is introduced for simulation iterative running, and its form is as follows: Cache the simulation process and simulation results;
[0094] Step 7.6: Determine whether the corresponding execution action has been executed for each second action information. If yes, proceed to step 7.7; otherwise, the remaining execution action sequence for each action information is executed continuously within a fixed step size, and return to step 7.5.
[0095] Step 7.7: Determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, output the cached result; otherwise, adjust the fixed step time and proceed to step 7.2.
[0096] On the other hand, the present invention provides a ship propulsion system equipment mission simulation system, comprising:
[0097] The power mission sequence construction module is used to create power mission sequences and supplement the power system mission sequence with the entity information of the power equipment in the combat method; wherein, the power system mission sequence is a mission sequence generated by randomly sorting the missions executed in the ship's power equipment.
[0098] The first environmental information pool construction module is used to establish a first environmental information pool based on the entity information of the power equipment in the combat method; wherein, the first environmental information pool is composed of random variables generated at the time of the occurrence of random events; the random event is the status of a certain mechanical part in the power system parts library during the execution of a mission;
[0099] The equipment-event queue construction module is used to load the entity information of each power equipment and the simulated combat mission of each power equipment for each piece of environmental information in the first environmental information pool, and to form equipment queues and event queues for each piece of equipment information and simulated combat missions respectively.
[0100] The second action information pool construction module is used to establish a second action information pool using an event queue.
[0101] The action queue construction module is used to allocate simulated combat tasks in the event queue to various second action information in the second action information pool, thereby obtaining the action queues of each second action information pool.
[0102] The current system simulation clock determination module is used to advance the simulation process using a fixed step time advance mechanism as the simulation clock. It uses an equipment queue to build a reward function to determine the size of the current system simulation clock. Simulated combat missions only exist within the fixed step time, and there are no simulated combat missions during the time interval.
[0103] The simulation execution module is used to simulate each simulated combat task in the event queue corresponding to each second action information based on the second action information pool and according to the current system simulation clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed.
[0104] More preferably, the method for obtaining the random variable includes the following steps:
[0105] Generate a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator;
[0106] The random number sequence is normalized using a modulus.
[0107] Random variables are obtained by using the inverse transformation method and function transformation method of the normalized random number sequence.
[0108] More preferably, the method by which the second action information pool construction module establishes the second action information pool is as follows:
[0109] Clustering tree diagram analysis is performed on the structural hierarchy of simulated combat mission information to establish mission logic. Each action information in the second action information pool is digitally described by the mission profile and the mission stage corresponding to each mission profile.
[0110] The information for each action is as follows: 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm. To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile includes air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile; the number of second action information items in the second action information pool is...
[0111] Where, N J The number of task profiles J; The number of task phases;
[0112] The second action information pool contains four action sets: air combat set, anti-ship attack set, anti-submarine attack set, and land attack set.
[0113] More preferably, the simulation execution module includes:
[0114] An initial event setting unit is used to set the initial events in the power system task sequence to a random state.
[0115] The current system simulation time determination unit is used to determine whether the current system simulation time is within a fixed step time based on the power system task sequence. If so, it obtains the current system simulation clock and sets the end time of the fixed step time corresponding to the current system simulation time as the current system simulation time. It also determines whether the current system simulation time is less than the end time of the current simulated combat mission based on each second action information. If not, it moves the current simulated combat mission to the historical event queue.
[0116] The simulated combat mission acquisition unit is used to acquire the simulated combat mission corresponding to the current system simulation clock from the execution action queue corresponding to each second action information.
[0117] The simulation unit is used to execute the action of each second action information and interact with different environmental information in the first environmental information pool to obtain sample sequence and reward sequence. The state-action value function in the Q-learning algorithm is introduced to perform simulation iterative operation, and the simulation process and simulation results are cached.
[0118] The execution determination unit for each action is used to determine whether the corresponding action has been executed based on each second action information. If not, the remaining action sequence corresponding to each second action information is executed continuously within a fixed step size, driving the simulation unit to execute.
[0119] The execution determination unit of the second action information pool is used to determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, the cached result is output; otherwise, the fixed step time is adjusted to drive the determination unit of the current system simulation time to run.
[0120] Example
[0121] like Figure 1 As shown, this invention provides a method for establishing a mission model of a ship propulsion system based on Q-learning, including the following steps:
[0122] Step 1: Create a sequence of tasks for the dynamic system;
[0123] Before starting simulation modeling, we first establish a preliminary Q-table format and formulate a task sequence;
[0124] Step 2: Add the entity information of the power equipment in the combat plan to the power system mission sequence;
[0125] The combat methods select the combat mission to be simulated and extract the equipment entity information of the combat mission to be simulated to supplement the power system mission sequence of this embodiment;
[0126] Step 3: Based on the power system task sequence and the power equipment information in the combat method, establish the first environmental information pool;
[0127] In this embodiment, the power system task sequence establishes a first environmental information pool. The number of environmental information items in the first environmental information pool is represented by random numbers uniformly distributed within the interval [0,1] generated by a linear congruent generator.
[0128] X n+1 =(aX n +c)(modm)n≥0
[0129] Where m is the modulus, a is the multiplier, and c is the increment; the initial value X0 is the seed, and m > 0, m > a, m > c, and X0 < m; the calculation is performed using the congruence multiplication method;
[0130] Normalize the integer random number sequence using the modulus:
[0131]
[0132] Obtain a random number R in the interval [0,1]. n (n = 1, 2, ...), where 0 ≤ X n ≤(m-1);
[0133] The power equipment entity in this embodiment includes various types of power generating devices and transmission devices such as main engine, thruster, propulsion shaft system, fuel pipeline system, cooling pipeline system, and exhaust pipeline system;
[0134] Step 4: Load the basic information of each power equipment and the simulated combat missions involving that equipment into each piece of environmental information in the first environmental information pool, and form the equipment queue and event queue for each piece of equipment information and simulated combat missions respectively.
[0135] The power system task sequence of this embodiment loads the combat method task into the first environmental information pool for each environmental information item, and forms the entity information of each equipment entity and the simulated combat task into the equipment queue and the event queue respectively.
[0136] The entity information in this embodiment includes equipment number, equipment model, equipment name, equipment status (normal, faulty), reliability distribution function type, and maintainability distribution function type, etc.
[0137] The simulated combat missions in this embodiment include air combat, anti-ship attack, anti-submarine attack, and land attack; the simulated combat mission information includes start and end times, combat start trigger mode, combat start trigger parameters, combat end trigger mode, and combat end trigger parameters.
[0138] Step 5: Establish a second action information pool using the task sequence of the power system;
[0139] The number of each action information item in the second action information pool is:
[0140] 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm.
[0141] The number of second action information items in the second action information pool is:
[0142] Step 6: Use the power system task sequence to assign the simulated combat tasks in the event queue to each second action information in the second action information pool, and obtain the execution action queue of each second action information pool;
[0143] Anti-ship attack: The first simulated combat mission is transit and rendezvous, and the second simulated combat mission is tactical deployment; Anti-air combat: The first simulated combat mission is turret repositioning, and the second simulated combat mission is attack;
[0144] Step 7: Use a fixed-step long-duration advancement mechanism based on the dynamic system task sequence as the simulation clock to advance the simulation process; adopt the fixed-step long-duration advancement mechanism to conform to the characteristics of reinforcement learning sequence decision-making, introduce reinforcement learning algorithm, and start advancing the simulation process;
[0145] Step 8: Construct a reward function using the task sequence of the dynamic system to determine the size of the current system simulation global clock TIME;
[0146] Anti-ship attack: The first simulated combat mission is transit and rendezvous, with a start time of 500 seconds and an end time of 2000 seconds; the second simulated mission is tactical deployment, with a start time of 700 seconds and an end time of 1300 seconds.
[0147] Air combat: The first simulated combat mission is turret repositioning, with a start time of 900 seconds and an end time of 1000 seconds; the second simulated mission is attack, with a start time of 1400 seconds and an end time of 3000 seconds.
[0148] By taking the union of the start and end times of the above activities, and using fixed step lengths of 500 seconds, 1000 seconds, 1500 seconds and 2000 seconds, the simulation clock is advanced in four steps.
[0149] The reward function formula is:
[0150]
[0151] Where, p t Represents time priority; p m Represents task priority, and p m The value range is [0.2, 0.4, 0.6, 0.8]. k is the weight parameter; λ is the time required for the event to execute.
[0152] Step 9: Using the second action information, perform simulation on each simulated combat task in the event queue corresponding to each second action information according to the current system simulation clock size, until each second action information has completed the execution of the corresponding execution action queue representing each combat task event;
[0153] In this embodiment, each piece of action information in the second action information pool is selected by the agent from each executable action sequence, and continuously interacts with the environmental information, specifically including the following steps:
[0154] Step 9.1: Set the initial events in the dynamic system task sequence to a blank state;
[0155] Step 9.2: Use the dynamic system task sequence to determine whether the current simulation time is within a fixed step time. If so, obtain the current system simulation clock and proceed to step 9.3; otherwise, obtain the corresponding combat task for each second action information from its respective execution action queue under the current simulation clock and proceed to step 9.4.
[0156] If we advance forward in a fixed time step of 1000 seconds, we have two executable actions: tactical deployment and turret repositioning. Within this time step, we can explore and utilize the ε-greedy algorithm to perform these actions.
[0157] Step 9.3: Use the dynamic system task sequence to set the end time of the fixed step period corresponding to the current simulation time to the current simulation time, and return to step 9.2;
[0158] Step 9.4: Determine whether the current time is less than the end time of the current simulated combat mission based on the information of each second action. If so, proceed to step 9.5; otherwise, move the current simulated combat mission to the historical event queue and proceed to step 9.6.
[0159] If the time advances in a fixed time step of 1500 seconds and the attack mission time is 1600 seconds, then the action to be executed will be temporarily removed from the historical event queue.
[0160] Step 9.5: Interact each second action information with different environmental information in the first environmental information pool to obtain a large number of sample sequences. The corresponding return sequence is The state-action value function from the Q-learning algorithm is introduced for simulation iterative running, and its form is as follows: Cache the simulation process and simulation results;
[0161] Step 9.6: Determine whether the corresponding execution action has been executed for each second action information. If yes, proceed to step 9.7; otherwise, the remaining execution action sequence for each action information is executed continuously within a fixed step size, and return to step 9.5.
[0162] The formula for Q-learning is the TD method representation of the Bellman formula, and the specific formula is as follows:
[0163]
[0164] This formula is used to establish the algorithm logic and perform computer modeling and simulation.
[0165] Step 9.7: Determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, output the cached result; otherwise, adjust the fixed step time and proceed to step 9.2.
[0166] Guided by the reward function of the Markov decision model, the system continues to execute the action queue for each action information once, obtaining the change in equipment information state and the immediate reward after each combat mission is selected and executed. This allows for real-time updates and optimization of the power equipment mission sequence. Furthermore, with the advancement of the simulation clock, the system obtains the corresponding start interval and allowable delay time for combat missions between mission profiles in the mission sequence. Based on this, the relationship between missions and equipment is established.
[0167] This invention proposes a Q-learning-based method for establishing ship propulsion system equipment task models, which integrates features such as sequential decision-making, ε-greedy greedy rules, and off-policy strategy evaluation methods based on the concept of optimal value. This method forms a complete set of theoretical methods for establishing ship propulsion system equipment task models and is the first to apply reinforcement learning to ship RMS simulation design.
[0168] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for simulating the mission of a ship's propulsion system, characterized in that, Includes the following steps: D1: Create a power system task sequence and supplement the power system task sequence with the entity information of the power equipment in the combat method; wherein, the power system task sequence is a task sequence generated by randomly sorting the tasks executed in the ship's power equipment; D2: Based on the entity information of the power equipment in the combat method, establish a first environmental information pool; wherein, the first environmental information pool is composed of random variables generated at the moment of the occurrence of a random event; the random event is the status of a certain mechanical part in the power system parts library during the execution of a mission; D3: Load the entity information of each power equipment and the simulated combat mission of each power equipment for each environmental information item in the first environmental information pool, and form the equipment information and simulated combat mission into equipment queue and event queue respectively. D4: Establish a second action information pool using an event queue; D5: Assign the simulated combat missions in the event queue to the various second action information in the second action information pool to obtain the execution action queue of each second action information pool; D6: The simulation process is advanced by using a fixed step time duration mechanism as the simulation clock. A reward function is built using an equipment queue to determine the current global simulation clock size. Simulated combat missions only exist within the fixed step time duration, and there are no simulated combat missions during the time interval. D7: Based on the second action information in the second action information pool, perform simulation on each simulated combat task in the event queue corresponding to each second action information according to the current system simulation global clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed.
2. The ship propulsion system equipment mission simulation method according to claim 1, characterized in that, The method for obtaining the random variable includes the following steps: Generate a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator; The random number sequence is normalized using a modulus. Random variables are obtained by using the inverse transformation method and function transformation method of the normalized random number sequence.
3. The ship propulsion system equipment mission simulation method according to claim 1 or 2, characterized in that, The method for establishing a second action information pool using simulated combat mission information is as follows: Clustering tree diagram analysis is performed on the structural hierarchy of simulated combat mission information to establish mission logic. Each action information in the second action information pool is digitally described by the mission profile and the mission stage corresponding to each mission profile. The information for each action is as follows: 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm. To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile includes air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile; the number of second action information items in the second action information pool is... Where, N J The number of task profiles J; The number of mission phases; the second action information pool contains four action sets, namely air combat set, anti-ship attack set, anti-submarine attack set and land attack set.
4. The ship propulsion system equipment mission simulation method according to claim 3, characterized in that, D7 specifically includes the following steps: D7.1: Set the initial events in the dynamic system task sequence to random states; D7.2: Based on the task sequence of the dynamic system, determine whether the current simulation time is within a fixed step time. If so, obtain the current system simulation clock and proceed to D7.3; otherwise, obtain the simulation combat task corresponding to the current system simulation clock from the execution action queue corresponding to each second action information and proceed to D7.
4. D7.3: Set the end time of the fixed step time corresponding to the current system simulation time to the current system simulation time, then go to D7.2; D7.4: Determine whether the current system simulation time is less than the end time of the current simulated combat mission based on the information of each second action. If so, proceed to D7.5; otherwise, move the current simulated combat mission to the historical event queue and proceed to D7.
6. D7.5: The second action information is executed and interacted with different environmental information in the first environmental information pool to obtain sample sequence and reward sequence. The state-action value function in the Q-learning algorithm is introduced to perform simulation iterative operation, and the simulation process and simulation results are cached. D7.6: Determine whether the corresponding execution action has been executed for each second action information. If yes, proceed to D7.7; otherwise, the remaining execution action sequence for each second action information is executed continuously within a fixed step size, and return to D7.
5. D7.7: Determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, output the cached result; otherwise, adjust the fixed step time and proceed to step 7.
2.
5. A ship propulsion system equipment mission simulation system, characterized in that, include: The power mission sequence construction module is used to create power mission sequences and supplement the power system mission sequence with the entity information of the power equipment in the combat method; wherein, the power system mission sequence is a mission sequence generated by randomly sorting the missions executed in the ship's power equipment. The first environmental information pool construction module is used to establish a first environmental information pool based on the entity information of the power equipment in the combat method; wherein, the first environmental information pool is composed of random variables generated at the time of the occurrence of random events; the random event is the status of a certain mechanical part in the power system parts library during the execution of a mission; The equipment-event queue construction module is used to load the entity information of each power equipment and the simulated combat mission of each power equipment for each piece of environmental information in the first environmental information pool, and to form equipment queues and event queues for each piece of equipment information and simulated combat missions respectively. The second action information pool construction module is used to establish a second action information pool using an event queue. The action queue construction module is used to allocate simulated combat tasks in the event queue to various second action information in the second action information pool, thereby obtaining the action queues of each second action information pool. The current system simulation clock determination module is used to advance the simulation process using a fixed step time advance mechanism as the simulation clock. It uses an equipment queue to build a reward function to determine the size of the current system simulation clock. Simulated combat missions only exist within the fixed step time, and there are no simulated combat missions during the time interval. The simulation execution module is used to simulate each simulated combat task in the event queue corresponding to each second action information based on the second action information pool and according to the current system simulation clock size, until all combat task events represented by the execution action queue corresponding to each second action information are executed.
6. The ship propulsion system equipment mission simulation system according to claim 5, characterized in that, The method for obtaining random variables includes the following steps: Generate a uniformly distributed random number sequence within the interval [0,1] using a linear congruential generator; The random number sequence is normalized using a modulus. Random variables are obtained by using the inverse transformation method and function transformation method of the normalized random number sequence.
7. The ship propulsion system equipment mission simulation system according to claim 5 or 6, characterized in that, The method for the second action information pool construction module to establish the second action information pool is as follows: Clustering tree diagram analysis is performed on the structural hierarchy of simulated combat mission information to establish mission logic. Each action information in the second action information pool is digitally described by the mission profile and the mission stage corresponding to each mission profile. The information for each action is as follows: 'a' represents each action information and also represents the action to be executed in the action space based on the Markov Decision Process in the Q-learning algorithm. To extract a task from task profile J; To start from mission phase J K Select one mission phase; the mission profile includes air combat, anti-ship attack, anti-submarine attack, and land attack; the mission phase is the action phase for completing each mission profile; the number of second action information items in the second action information pool is... Where, N J The number of task profiles J; The number of task phases; The second action information pool contains four action sets: air combat set, anti-ship attack set, anti-submarine attack set, and land attack set.
8. The ship propulsion system equipment mission simulation system according to claim 7, characterized in that, The simulation execution module includes: An initial event setting unit is used to set the initial events in the power system task sequence to a random state. The current system simulation time determination unit is used to determine whether the current system simulation time is within a fixed step time based on the power system task sequence. If so, it obtains the current system simulation clock and sets the end time of the fixed step time corresponding to the current system simulation time as the current system simulation time. It also determines whether the current system simulation time is less than the end time of the current simulated combat mission based on each second action information. If not, it moves the current simulated combat mission to the historical event queue. The simulated combat mission acquisition unit is used to acquire the simulated combat mission corresponding to the current system simulation clock from the execution action queue corresponding to each second action information. The simulation unit is used to execute the action of each second action information and interact with different environmental information in the first environmental information pool to obtain sample sequence and reward sequence. The state-action value function in the Q-learning algorithm is introduced to perform simulation iterative operation, and the simulation process and simulation results are cached. The execution determination unit for each action is used to determine whether the corresponding action has been executed based on each second action information. If not, the remaining action sequence corresponding to each second action information is executed continuously within a fixed step size, driving the simulation unit to execute. The execution determination unit of the second action information pool is used to determine whether all actions in the second action information pool have been executed based on the task sequence of the power system. If so, the cached result is output; otherwise, the fixed step time is adjusted to drive the determination unit of the current system simulation time to run.