A method and device for adaptive planning of ship tasks based on deep reinforcement learning

Abstract

The present invention discloses a method and device for adaptive planning of ship tasks based on deep reinforcement learning, the method comprising: obtaining ship task planning data information; the ship task planning data information comprises historical data information and personnel planning data information; preprocessing the ship task planning data information to obtain preprocessed data information; using the preprocessed data information, training a preset deep reinforcement learning planning model to obtain an optimized deep reinforcement learning planning model; using the optimized deep reinforcement learning planning model, processing the ship task data information to be processed to obtain a ship task planning scheme. By introducing a deep reinforcement learning algorithm, the present invention can more effectively process these complex data, extract valuable features and patterns therefrom, and thus provide more accurate and comprehensive data support for task planning.

Description

Technical Field

[0001] The present invention relates to the technical field of ship mission planning, and in particular to a ship mission adaptive planning method and device based on deep reinforcement learning. Background Art

[0002] As the complexity and variability of modern environments increase, ships face unprecedented challenges in performing their missions. Traditional mission planning methods, such as rule-based or simple machine learning methods, have been unable to meet the adaptive needs in complex environments. These methods often lack flexibility and have difficulty adapting to rapidly changing environments and mission requirements. They also have limitations when processing high-dimensional, nonlinear data. Therefore, it is particularly important to develop an intelligent mission planning system that can make full use of historical experience data, adapt to environmental changes in real time, and provide personalized decision support for ship personnel.

[0003] Existing technologies for ship mission planning have several significant shortcomings that limit the effectiveness and adaptability of the system in complex and dynamic environments. Specifically:

[0004] Limited data processing capabilities: Traditional methods often have difficulty processing high-dimensional, nonlinear, and time-series data, and are unable to fully extract valuable information from rich historical experience.

[0005] Lack of real-time adaptability: Faced with rapidly changing environmental and task conditions, existing systems are often unable to adjust strategies in real time, resulting in delayed and inaccurate decisions.

[0006] Insufficient personalized support: Existing technologies rarely take into account the individual differences and skill levels of ship personnel and are unable to provide targeted decision support, thus affecting overall combat effectiveness.

[0007] Difficulty in model updating: When new data or new situations emerge, traditional models may need to be retrained or extensively adjusted, which is time-consuming and laborious in practice.

[0008] Limited collaboration capabilities: In missions that require multi-ship or multi-department collaboration, existing technologies often lack effective coordination and communication mechanisms. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a method and device for adaptive planning of ship tasks based on deep reinforcement learning, combining historical experience data and personnel data, and training and optimizing decision models through complex deep reinforcement learning algorithms. In this way, not only efficient task planning can be achieved, but also personalized action suggestions can be provided according to individual differences of personnel, thereby improving the execution efficiency and success rate of the overall task.

[0010] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a ship task adaptive planning method based on deep reinforcement learning, the method comprising:

[0011] S1, obtaining ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information;

[0012] S2, preprocessing the ship mission planning data information to obtain preprocessed data information;

[0013] S3, using the preprocessed data information to train a preset deep reinforcement learning planning model to obtain an optimized deep reinforcement learning planning model;

[0014] S4, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain a ship mission planning solution.

[0015] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the preprocessing of the ship mission planning data information to obtain the preprocessed data information includes:

[0016] S21, performing data cleaning on the ship mission planning data information to obtain cleaned data information;

[0017] S22, performing data conversion on the cleaning data information to obtain standard data information;

[0018] S23, extracting features from the standard data information to obtain pre-processed data information.

[0019] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the feature extraction of the standard data information to obtain the preprocessed data information includes:

[0020] S231, extracting features from the standard data information to obtain first feature information;

[0021] S232, extracting features from the standard data information to obtain second feature information;

[0022] S233, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0023] S234, performing mapping interpolation processing on the second feature information to obtain second mapping feature information;

[0024] S235: Process the first mapping feature information and the second mapping feature information to obtain preprocessing data information.

[0025] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the extracting features of the standard data information to obtain the first feature information includes:

[0026] Using a preset first feature extraction model, extracting features from the standard data information to obtain first feature information;

[0027] The first feature extraction model expression is:

[0028]

[0029] Among them, C s (t,f) is the first feature information, f is frequency, t is time, f(ξ,τ) represents the kernel function; s(t) is the input standard data information, τ is the time shift, ξ is the frequency shift, u is the integral variable, exp is the exponential function, σ is a constant, and * indicates taking the conjugate.

[0030] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the performing mapping interpolation processing on the first feature information to obtain the first mapping feature information includes:

[0031] Using a preset mapping interpolation model, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0032] The preset mapping interpolation model expression is:

[0033]

[0034] Wherein, M is the first feature information, M′ is the first mapping feature information, and M i ' j is an element in M′, M min is the minimum value in M, M max is the maximum value in M, 1≤i≤m, 1≤j≤n, m, n are the row and column lengths of the matrix M.

[0035] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the first mapping feature information and the second mapping feature information to obtain preprocessed data information includes:

[0036] S2351, processing the first mapping feature information to obtain an autocovariance matrix Σ 11 ;

[0037] S2352, processing the second mapping feature information to obtain an autocovariance matrix Σ 22 ;

[0038] S2353, processing the first mapping feature information and the second mapping feature information to obtain a first cross-covariance matrix ∑ 12 and the second cross-covariance matrix ∑ 21 ;

[0039] S2354, solving the preset optimization objective function to obtain the transformation matrix W x and W y ;

[0040] The preset optimization objective function is:

[0041]

[0042] Where Q1 = Σ 11 , Q2=Σ 22 , β, λ, θ are coefficient constants, and T is the transpose;

[0043] S2355, the transformation matrix W x and W y Processing is performed to obtain preprocessed data information.

[0044] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the optimized deep reinforcement learning planning model is used to process the ship mission data information to be processed to obtain a ship mission planning scheme, including:

[0045] S41, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain decision information;

[0046] S42, performing data processing on the decision information and the personnel planning data information to obtain a ship mission planning solution.

[0047] A second aspect of an embodiment of the present invention discloses a ship task adaptive planning device based on deep reinforcement learning, the device comprising:

[0048] An information acquisition module is used to acquire ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information;

[0049] A preprocessing module, used for preprocessing the ship mission planning data information to obtain preprocessed data information;

[0050] A model training module, used to train a preset deep reinforcement learning planning model using the preprocessed data information to obtain an optimized deep reinforcement learning planning model;

[0051] The ship mission planning module is used to process the ship mission data information to be processed using the optimized deep reinforcement learning planning model to obtain a ship mission planning solution.

[0052] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the preprocessing of the ship mission planning data information to obtain the preprocessed data information includes:

[0053] S21, performing data cleaning on the ship mission planning data information to obtain cleaned data information;

[0054] S22, performing data conversion on the cleaning data information to obtain standard data information;

[0055] S23, extracting features from the standard data information to obtain pre-processed data information.

[0056] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the feature extraction of the standard data information to obtain the preprocessed data information includes:

[0057] S231, extracting features from the standard data information to obtain first feature information;

[0058] S232, extracting features from the standard data information to obtain second feature information;

[0059] S233, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0060] S234, performing mapping interpolation processing on the second feature information to obtain second mapping feature information;

[0061] S235: Process the first mapping feature information and the second mapping feature information to obtain preprocessing data information.

[0062] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the extracting features of the standard data information to obtain the first feature information includes:

[0063] Using a preset first feature extraction model, extracting features from the standard data information to obtain first feature information;

[0064] The first feature extraction model expression is:

[0065]

[0066] Among them, C s(t,f) is the first feature information, f is frequency, t is time, f(ξ,τ) represents the kernel function; s(t) is the input standard data information, τ is the time shift, ξ is the frequency shift, u is the integral variable, exp is the exponential function, σ is a constant, and * indicates taking the conjugate.

[0067] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the performing mapping interpolation processing on the first feature information to obtain the first mapping feature information includes:

[0068] Using a preset mapping interpolation model, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0069] The preset mapping interpolation model expression is:

[0070]

[0071] Wherein, M is the first feature information, M′ is the first mapping feature information, and M i ' j is an element in M′, M min is the minimum value in M, M max is the maximum value in M, 1≤i≤m, 1≤j≤n, m, n are the row and column lengths of the matrix M.

[0072] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the first mapping feature information and the second mapping feature information to obtain preprocessed data information includes:

[0073] S2351, processing the first mapping feature information to obtain an autocovariance matrix Σ 11 ;

[0074] S2352, processing the second mapping feature information to obtain an autocovariance matrix Σ 22 ;

[0075] S2353: Process the first mapping feature information and the second mapping feature information to obtain a first cross-covariance matrix Σ 12 and the second cross-covariance matrix ∑ 21 ;

[0076] S2354, solving the preset optimization objective function to obtain the transformation matrix W x and W y ;

[0077] The preset optimization objective function is:

[0078]

[0079] Where Q1 = Σ 11 , Q2=Σ 22 , β, λ, θ are coefficient constants, and T is the transpose;

[0080] S2355, the transformation matrix W x and W y Processing is performed to obtain preprocessed data information.

[0081] As an optional implementation, in the second aspect of the embodiment of the present invention, the optimized deep reinforcement learning planning model is used to process the ship mission data information to be processed to obtain a ship mission planning scheme, including:

[0082] S41, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain decision information;

[0083] S42, performing data processing on the decision information and the personnel planning data information to obtain a ship mission planning solution.

[0084] The third aspect of the present invention discloses another device for adaptive planning of ship tasks based on deep reinforcement learning, the device comprising:

[0085] A memory storing executable program code;

[0086] a processor coupled to the memory;

[0087] The processor calls the executable program code stored in the memory to execute part or all of the steps in the ship task adaptive planning method based on deep reinforcement learning disclosed in the first aspect of an embodiment of the present invention.

[0088] The fourth aspect of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps in the ship task adaptive planning method based on deep reinforcement learning disclosed in the first aspect of an embodiment of the present invention.

[0089] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0090] (1) Improve data processing capabilities: In the prior art, ship mission planning systems often have difficulty processing high-dimensional, nonlinear, and time-series data, resulting in the inability to fully utilize rich historical experience data to optimize decision-making. The present invention introduces a deep reinforcement learning algorithm to more effectively process these complex data, extract valuable features and patterns from them, and provide more accurate and comprehensive data support for mission planning.

[0091] (2) Enhanced real-time adaptability: Current systems usually lack the ability to adjust strategies in real time when facing rapidly changing environments and task conditions, resulting in delayed and inaccurate decisions. This invention uses a real-time feedback mechanism and online learning capabilities to enable the system to dynamically adjust strategies according to real-time conditions during task execution, ensuring the timeliness and accuracy of decision-making.

[0092] (3) Personalized decision support: Existing technologies rarely consider the individual differences and skill levels of ship personnel and cannot provide targeted decision support. The present invention combines personnel data, including individual differences such as personnel skills and experience, to generate personalized action suggestions for each person, thereby maximizing the potential of personnel and improving overall combat efficiency.

[0093] (4) Simplify the model update process: Traditional models often need to be retrained or adjusted a lot when facing new data or new situations, which is not only time-consuming and labor-intensive, but may also affect the stability of the model. The model design of the present invention takes into account the needs of continuous learning and incremental updates, and can quickly and effectively update the model when new data arrives without retraining the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0095] Figure 1 It is a flowchart of a method for adaptive planning of ship tasks based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0096] Figure 2 It is a flowchart of another method for adaptive planning of ship tasks based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0097] Figure 3 is a deep reinforcement learning flow chart disclosed in an embodiment of the present invention;

[0098] Figure 4 It is a structural schematic diagram of a ship task adaptive planning device based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0099] Figure 5 It is a structural schematic diagram of another ship task adaptive planning device based on deep reinforcement learning disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0100] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0102] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0103] The present invention discloses a method and device for adaptive planning of ship tasks based on deep reinforcement learning, the method comprising: obtaining ship task planning data information; the ship task planning data information comprises historical data information and personnel planning data information; preprocessing the ship task planning data information to obtain preprocessed data information; using the preprocessed data information, training a preset deep reinforcement learning planning model to obtain an optimized deep reinforcement learning planning model; using the optimized deep reinforcement learning planning model, processing the ship task data information to be processed to obtain a ship task planning scheme. The following are detailed descriptions respectively.

[0104] Embodiment 1

[0105] See also Figure 1 , Figure 1 is a flow chart of a method for adaptive planning of ship tasks based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 1 The described method for adaptive ship mission planning based on deep reinforcement learning is applied to the field of ship mission planning technology, and the embodiments of the present invention are not limited thereto. Figure 1As shown, the ship mission adaptive planning method based on deep reinforcement learning may include the following operations:

[0106] S1, obtaining ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information;

[0107] S2, preprocessing the ship mission planning data information to obtain preprocessed data information;

[0108] S3, using the preprocessed data information to train a preset deep reinforcement learning planning model to obtain an optimized deep reinforcement learning planning model;

[0109] S4, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain a ship mission planning solution.

[0110] Optionally, the preprocessing the ship mission planning data information to obtain preprocessed data information includes:

[0111] S21, performing data cleaning on the ship mission planning data information to obtain cleaned data information;

[0112] Removing outliers from the ship mission planning data information and interpolating missing values ​​to obtain cleaned data information;

[0113] Abnormal values ​​are data information whose average value is greater than a preset threshold;

[0114] Missing values ​​are missing information in the ship mission planning data information, and the missing values ​​are obtained by comparing with the empirical information;

[0115] S22, performing data conversion on the cleaning data information to obtain standard data information;

[0116] The cleaned data information is normalized and divided by the maximum value, and the obtained standard data information belongs to [0, 1].

[0117] S23, extracting features from the standard data information to obtain pre-processed data information.

[0118] Optionally, the extracting features from the standard data information to obtain pre-processed data information includes:

[0119] S231, extracting features from the standard data information to obtain first feature information;

[0120] S232, extracting features from the standard data information to obtain second feature information;

[0121] S233, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0122] S234, performing mapping interpolation processing on the second feature information to obtain second mapping feature information;

[0123] S235: Process the first mapping feature information and the second mapping feature information to obtain preprocessing data information.

[0124] Optionally, the extracting features from the standard data information to obtain first feature information includes:

[0125] Using a preset first feature extraction model, extracting features from the standard data information to obtain first feature information;

[0126] The first feature extraction model expression is:

[0127]

[0128] Among them, C s (t,f) is the first feature information, f is frequency, t is time, f(ξ,τ) represents the kernel function; s(t) is the input standard data information, τ is the time shift, ξ is the frequency shift, u is the integral variable, exp is the exponential function, σ is a constant, and * indicates taking the conjugate.

[0129] Optionally, performing mapping interpolation processing on the first feature information to obtain first mapping feature information includes:

[0130] Using a preset mapping interpolation model, performing mapping interpolation processing on the first feature information to obtain first mapping feature information;

[0131] The preset mapping interpolation model expression is:

[0132]

[0133] Wherein, M is the first feature information, M′ is the first mapping feature information, and M i ' j is an element in M′, M min is the minimum value in M, M max is the maximum value in M, 1≤i≤m, 1≤j≤n, m, n are the row and column lengths of the matrix M.

[0134] Optionally, the processing the first mapping feature information and the second mapping feature information to obtain preprocessed data information includes:

[0135] S2351, processing the first mapping feature information to obtain an autocovariance matrix Σ 11 ;

[0136] S2352, processing the second mapping feature information to obtain an autocovariance matrix Σ 22 ;

[0137] S2353: Process the first mapping feature information and the second mapping feature information to obtain a first cross-covariance matrix Σ 12 and the second cross-covariance matrix ∑ 21 ;

[0138] S2354, solving the preset optimization objective function to obtain the transformation matrix W x and W y ;

[0139] The preset optimization objective function is:

[0140]

[0141] Where Q1 = Σ 11 , Q2=Σ 22 , β, λ, θ are coefficient constants, and T is the transpose;

[0142] S2355, the transformation matrix W x and W y Processing is performed to obtain preprocessed data information.

[0143] The preprocessed data information is:

[0144]

[0145] z is the preprocessed data information, X1 is the first mapping feature information, and X2 is the second mapping feature information.

[0146] Extracting features from the standard data information to obtain second feature information includes:

[0147] The standard data information is decomposed into IMFs with different time scales, simple components and relatively stable. All the maximum and minimum points of x'(t) are determined, and the corresponding upper envelope e is obtained by polynomial interpolation. max (t) and the lower envelope e min (t), and the mean m(t)=(e min (t)+e max(t)) / 2, extract the detail h(t) according to x'(t) and the mean m(t), let h(t) = x'(t)-m(t), if h(t) conforms to IMF, it is the first modal component IMF1 (denoted as c1(t)); otherwise let h(t) = x'(t), continue to decompose, and finally decompose into Among them, r(t) is the remainder. The normalization function is Among them, x max is the maximum value of data information, x min is the minimum value of the data information, the data information is x(t), and x'(t) is the normalized data information.

[0148] Optionally, the optimized deep reinforcement learning planning model is used to process the ship mission data information to be processed to obtain a ship mission planning scheme, including:

[0149] S41, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain decision information;

[0150] S42, performing data processing on the decision information and the personnel planning data information to obtain a ship mission planning solution.

[0151] The deep reinforcement learning planning model is multi-reward reinforcement learning (MRRL), which is a generalization of standard single-reward reinforcement learning. Therefore, it is also based on the Markov decision process as its framework. Its composition is (S, A, P, R, γ), where R is the reward function, A is the action space, S is the state space, γ∈(0,1] is the discount factor, and P is the state transition probability.

[0152] The strategy π is:

[0153] π(a|s)=p[A t =a|S t =s]

[0154] As shown in the formula, choosing action a when the state is s is the formulation of a strategy.

[0155] However, unlike single-reward reinforcement learning, the reward function of MRRL does not return a scalar value, but returns a scalar vector that reflects each reward value m as a result.

[0156] R(s,a,a')=(R1(s,a,s'),…,R m (s,a,s'))

[0157] The strategy in this case is also determined by a vector of expected values:

[0158]

[0159] The goal of multi-reward reinforcement learning is to find the optimal decision for the entire system, or to approximate the optimal decision for the entire system. In the process of solving multi-reward reinforcement learning problems, when using the Q-learning reinforcement learning method, the Q value of each model can be learned in parallel and stored in the form of a vector:

[0160]

[0161] The most common way to derive a policy from these estimates is to compute a linear scalar or weighted sum over the Q vector and the weight vector w:

[0162]

[0163] The weight vector w represents the proportion of each objective function. For the multi-reward problem, whichever reward function should be solved first will have a larger weight value. However, it is difficult and often unscientific to achieve the trade-off of multiple rewards by setting these weight values ​​a priori. Usually, a large number of parameters are required for adjustment.

[0164] Embodiment 2

[0165] See also Figure 2 , Figure 2 is a flow chart of another method for adaptive planning of ship tasks based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 2 The described method for adaptive ship mission planning based on deep reinforcement learning is applied to the field of ship mission planning technology, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the ship mission adaptive planning method based on deep reinforcement learning may include the following operations:

[0166] The purpose of the present invention is to provide a ship mission adaptive planning system based on deep reinforcement learning to solve the defects of the prior art, such as limited data processing capability, lack of real-time adaptability, inability to provide personalized decision-making, and insufficient multi-agent collaboration capability.

[0167] To achieve the above object, the present invention adopts the following technical solutions:

[0168] The system structure includes data preprocessing module, deep reinforcement learning module, personalized decision support module and multi-personnel intelligent collaboration feedback module. The modules are interconnected through data flow and control flow to realize information transmission and processing. Data preprocessing module: responsible for extracting relevant data from the historical experience database and personnel database, and performing cleaning, conversion and feature extraction to meet the needs of deep reinforcement learning algorithms.

[0169] Ship tasks include navigation tasks, cargo loading and unloading tasks, ship maintenance tasks, etc. The task type is determined according to the task request and passed to the corresponding processing module; key features are extracted including ship status, environmental parameters, task-related features, etc.

[0170] Deep reinforcement learning module: uses complex deep reinforcement learning algorithms, such as PPO or A2C, combined with simulators for model training. This module is able to handle high-dimensional state space and action space, and autonomously learns and improves through interaction with the environment.

[0171] Personalized decision support module: Combines personnel data, such as skills, experience, etc., to generate personalized action recommendations for each person. This module displays recommendations through a user interface and collects feedback from personnel to achieve personalized decision support.

[0172] Multi-agent collaboration module: In situations where collaboration between multiple ships or departments is required, a multi-agent system is used to coordinate their actions. This module ensures information sharing and smooth communication to achieve efficient completion of the overall task.

[0173] Data preprocessing: Extract relevant data from the historical experience database and personnel database, including task execution records, environmental information, personnel skills, etc. Clean, transform and extract features from the data to obtain a data set suitable for deep reinforcement learning algorithms.

[0174] Data preprocessing is responsible for extracting relevant data from the historical experience database and personnel database, and performing cleaning, conversion, and feature extraction to meet the needs of deep reinforcement learning algorithms.

[0175] The historical experience database includes but is not limited to navigation logs, accident reports, and historical mission execution status, etc. These data cover the ship's operating experience and performance under different environmental conditions.

[0176] Data cleaning: Remove duplicate records, outliers, and incomplete data.

[0177] Data conversion: Convert data of different formats and units into a format suitable for deep reinforcement learning algorithm processing, such as converting time series data into fixed-length window data.

[0178] There are several feature extraction methods:

[0179] (1) Laplacian Eigenmaps and Gaussian-Kernel Density Estimation (G-KDE) are used to compress ship trajectory data and extract key turning points.

[0180] (2) The fuzzy adaptive DBSCAN method is used to cluster turning points and identify common turning areas.

[0181] (3) Extract ship features and task-related features as input to the deep reinforcement learning model.

[0182] Deep reinforcement learning: Use complex deep reinforcement learning algorithms such as PPO or A2C to train models. Simulators are built to recreate historical scenarios, and data augmentation techniques are used to expand the data set. The model continuously interacts with the environment during training and adjusts its strategy based on feedback to achieve autonomous learning and improvement.

[0183] Using complex deep reinforcement learning algorithms, such as PPO or A2C, combined with simulators for model training. This module is able to handle high-dimensional state space and action space, and autonomously learn and improve through interaction with the environment.

[0184] The simulator can simulate the navigation conditions of ships under different sea conditions, weather conditions and mission planning, etc., providing a rich training environment for deep reinforcement learning models.

[0185] The model training process is as follows:

[0186] Algorithm selection: Use deep reinforcement learning algorithms such as Proximal Policy Optimization (PPO) or Advantageous Actor-Critic (A2C).

[0187] Simulator construction: Build a simulator based on historical experience data and personnel data.

[0188] Training data: Use data from the historical experience database as training samples, including ship status, tasks, personnel, and corresponding action decisions and result feedback.

[0189] Training process: The preprocessed data is input into the deep reinforcement learning model, and multiple rounds of iterative training are performed through the simulator to continuously adjust the model parameters to optimize the decision-making strategy.

[0190] Personalized decision support: Combined with personnel data, personalized action suggestions are generated for each person. Suggestions are displayed through the user interface and feedback from personnel is collected. The system adjusts and optimizes the suggestions based on the feedback to provide decision support that better meets actual needs.

[0191] Personalized decision support combines personnel data, such as skills, experience, etc., to generate personalized action recommendations for each person. This module displays recommendations through a user interface and collects feedback from personnel to achieve personalized decision support.

[0192] The specific steps of personalized decision support are as follows:

[0193] (1) Personnel data analysis: Collect and analyze personnel data, including skill levels, historical operation records, preference settings, etc.

[0194] (2) Decision recommendation generation: Combine the output of the deep reinforcement learning model and personnel data to generate personalized action recommendations for each person.

[0195] (3) User interaction and feedback: Personalized suggestions are displayed through the user interaction interface, and real-time feedback from personnel is collected to further optimize the decision-making model.

[0196] (4) Model update: Dynamically adjust decision model parameters based on personnel feedback to achieve continuous optimization of personalized decisions.

[0197] Multi-personnel intelligent collaboration: When multiple ships or departments need to collaborate, a multi-agent system is used to coordinate their actions. The system ensures smooth information sharing and communication so that each ship or department can understand each other's status and intentions in real time, thereby achieving efficient completion of the overall task.

[0198] In situations where collaboration between multiple ships or departments is required, a multi-agent system is used to coordinate their actions. This module ensures information sharing and smooth communication to achieve efficient completion of the overall mission.

[0199] The specific implementation is as follows:

[0200] Agent definition: Each ship or key department is mapped as an agent, and each agent has an independent deep reinforcement learning model and decision-making ability.

[0201] Information sharing and communication: Build a multi-agent communication system to ensure that each agent can share status information, task progress, and decision results in real time.

[0202] Collaborative decision making: Use attention reasoning methods to extract key data that helps collaborative decision making, and design memory-driven experience learning methods to accumulate interactive experience. Introduce multi-head attention mechanisms and noise networks to enhance the exploration ability and robustness of intelligent agent decision making.

[0203] Conflict resolution and strategy optimization: When decision conflicts occur, they are resolved through negotiation mechanisms or preset rules to ensure efficient completion of the overall task. At the same time, based on feedback during task execution, the decision strategies of each agent are dynamically adjusted to achieve collaborative optimization closed-loop control.

[0204] Among them, deep reinforcement learning flow chart Figure 3 The model training process is demonstrated, including data input, feature extraction, model training, strategy output and other steps. Through continuous iteration and optimization, the model can gradually adapt to environmental changes and propose better decision-making strategies.

[0205] Environment initialization: The environment provides the agent with initial state data, which is the basis for understanding the current environment.

[0206] Feature extraction: After receiving the state data of the environment, feature extraction is performed to convert the raw data into feature vectors that the model can understand and process.

[0207] Model training: Based on the extracted features, train its deep reinforcement learning model. By adjusting the model's parameters, its ability to predict the best action based on the current state is continuously improved.

[0208] The dynamic programming equation is used in the implementation of deep reinforcement learning. The formula is as follows:

[0209]

[0210] The π(a|s) mentioned in the formula is indeed a probability distribution, which represents the probability of selecting action a given state s; q(s,a) is an action value function, which estimates the expected return that can be obtained by taking action a given state s.

[0211] (1) Action strategy output: After training is completed, the model will output an action strategy based on the characteristics of the current state.

[0212] (2) Environmental feedback: After an action is performed, the environment will give a corresponding reward signal and update to a new state. The new state provides the basis for the next decision.

[0213] (3) Model optimization: Optimize the model based on the reward signal and new state fed back by the environment.

[0214] Through continuous iteration, the model gradually adapts to the environment and improves the accuracy of decision-making.

[0215] (4) Iteration loop: The optimized model takes the new state as input and repeats the above steps to form a closed-loop iterative process. This process will continue until the model reaches a convergence state or meets the preset stop condition.

[0216] Embodiment 3

[0217] See also Figure 4 , Figure 4 : is a schematic diagram of a ship task adaptive planning device based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 4 The described ship mission adaptive planning device based on deep reinforcement learning is applied to the field of ship mission planning technology, and the embodiments of the present invention are not limited thereto. Figure 4 As shown, the ship task adaptive planning device based on deep reinforcement learning may include the following operations:

[0218] S301, an information acquisition module, used to acquire ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information;

[0219] S302, a preprocessing module, used to preprocess the ship mission planning data information to obtain preprocessed data information;

[0220] S303, a model training module, used to train a preset deep reinforcement learning planning model using the preprocessed data information to obtain an optimized deep reinforcement learning planning model;

[0221] S304, a ship mission planning module is used to process the ship mission data information to be processed using the optimized deep reinforcement learning planning model to obtain a ship mission planning solution.

[0222] Embodiment 4

[0223] See also Figure 5 , Figure 5 : is a schematic diagram of a ship task adaptive planning device based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 5 The described ship mission adaptive planning device based on deep reinforcement learning is applied to the field of ship mission planning technology, and the embodiments of the present invention are not limited thereto. Figure 5 As shown, the ship task adaptive planning device based on deep reinforcement learning may include the following operations:

[0224] A memory 401 storing executable program codes;

[0225] a processor 402 coupled to the memory 401;

[0226] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the ship task adaptive planning method based on deep reinforcement learning described in Example 1 and Example 2.

[0227] Embodiment 5

[0228] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the method for adaptive planning of ship tasks based on deep reinforcement learning described in Embodiment 1 and Embodiment 2.

[0229] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0230] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0231] Finally, it should be noted that the method and device for adaptive planning of ship tasks based on deep reinforcement learning disclosed in the embodiment of the present invention discloses only the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical schemes described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the various embodiments of the present invention.

Claims

1. A ship mission adaptive planning method based on deep reinforcement learning, characterized in that: The method comprises: S1, obtaining ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information; The historical data information includes navigation logs, accident reports and historical mission execution status; The personnel planning data information includes skill level, historical operation records and preference settings; S2, preprocessing the ship mission planning data information to obtain preprocessed data information; S3, using the preprocessed data information to train a preset deep reinforcement learning planning model to obtain an optimized deep reinforcement learning planning model; The pre-processed data information includes ship status, mission, personnel and corresponding action decisions and result feedback; S4, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain a ship mission planning solution, including: S41, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain decision information; S42, performing data processing on the decision information and the personnel planning data information to obtain a ship mission planning plan and generate personalized action suggestions for each person.

2. The method for adaptive ship mission planning based on deep reinforcement learning according to claim 1 is characterized in that: The preprocessing of the ship mission planning data information to obtain the preprocessed data information includes: S21, performing data cleaning on the ship mission planning data information to obtain cleaned data information; S22, performing data conversion on the cleaning data information to obtain standard data information; S23, extracting features from the standard data information to obtain pre-processed data information.

3. The method for adaptive ship mission planning based on deep reinforcement learning according to claim 2 is characterized in that: The step of extracting features from the standard data information to obtain pre-processed data information includes: S231, extracting features from the standard data information to obtain first feature information, including: Using a preset first feature extraction model, extracting features from the standard data information to obtain first feature information; The first feature extraction model expression is: Among them, C s (t,f) is the first feature information, f is frequency, t is time, f(ξ,τ) represents the kernel function; s(t) is the input standard data information, τ is the time shift, ξ is the frequency shift, u is the integral variable, exp is the exponential function, σ is a constant, and * indicates conjugation; S232, extracting features from the standard data information to obtain second feature information, including: performing EMD decomposition on the standard data information to decompose it into IMFs with different time scales, simple components, and relatively stable; S233, performing mapping interpolation processing on the first feature information to obtain first mapping feature information; S234, performing mapping interpolation processing on the second feature information to obtain second mapping feature information; S235: Process the first mapping feature information and the second mapping feature information to obtain preprocessing data information.

4. The method for adaptive ship mission planning based on deep reinforcement learning according to claim 3 is characterized in that: The performing mapping interpolation processing on the first feature information to obtain first mapping feature information includes: Using a preset mapping interpolation model, performing mapping interpolation processing on the first feature information to obtain first mapping feature information; The preset mapping interpolation model expression is: Wherein, M is the first feature information, M′ is the first mapping feature information, and M i ' j is an element in M′, M min is the minimum value in M, M max is the maximum value in M, 1≤i≤m, 1≤j≤n, m, n are the row and column lengths of the matrix M.

5. The method for adaptive ship mission planning based on deep reinforcement learning according to claim 3 is characterized in that: The processing of the first mapping characteristic information and the second mapping characteristic information to obtain preprocessed data information includes: S2351, processing the first mapping feature information to obtain an autocovariance matrix Σ 11 ; S2352, processing the second mapping feature information to obtain an autocovariance matrix Σ 22 ; S2353: Process the first mapping feature information and the second mapping feature information to obtain a first cross-covariance matrix Σ 12 and the second cross-covariance matrix Σ 21 ; S2354, solving the preset optimization objective function to obtain the transformation matrix W x and W y ; The preset optimization objective function is: Where Q1 = Σ 11 , Q2=Σ 22 , β, λ, θ are coefficient constants, and T is the transpose; S2355, the transformation matrix W x and W y Processing is performed to obtain preprocessed data information.

6. A ship mission adaptive planning device based on deep reinforcement learning, characterized in that: The device comprises: An information acquisition module is used to acquire ship mission planning data information; the ship mission planning data information includes historical data information and personnel planning data information; The historical data information includes navigation logs, accident reports and historical mission execution status; The personnel planning data information includes skill level, historical operation records and preference settings; A preprocessing module, used for preprocessing the ship mission planning data information to obtain preprocessed data information; A model training module, used to train a preset deep reinforcement learning planning model using the preprocessed data information to obtain an optimized deep reinforcement learning planning model; The pre-processed data information includes ship status, mission, personnel and corresponding action decisions and result feedback; The ship mission planning module is used to process the ship mission data information to be processed using the optimized deep reinforcement learning planning model to obtain a ship mission planning solution, including: S41, using the optimized deep reinforcement learning planning model to process the ship mission data information to be processed to obtain decision information; S42, performing data processing on the decision information and the personnel planning data information to obtain a ship mission planning plan and generate personalized action suggestions for each person.

7. A ship mission adaptive planning device based on deep reinforcement learning, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the ship task adaptive planning method based on deep reinforcement learning as described in any one of claims 1-5.

8. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, which, when called, are used to execute the ship task adaptive planning method based on deep reinforcement learning as described in any one of claims 1 to 5.

Images