Ship cooperative control method based on self-learning

Abstract

The present invention relates to the field of multi-agent collaborative technology, and discloses a ship collaborative control method based on self-learning, step S10, using an input module to obtain the parameters of each ship in the multi-agent system, including the expected operation trajectory of the ship; step S20, using a parameter and network initialization module, a network weight assignment module and a global loop module to start the global loop; step S30, using a network training module and a local loop module to start the local loop based on the initialized related parameters; step S40, using the iterative control law and the local optimal performance index function to replace the current iterative control law and performance index function, and using the output module to obtain the ship collaborative control strategy. In the present invention, combined with the key technologies of self-learning optimization ADP algorithm and multi-agent system communication, with the support of these technologies, it can be realized that multiple ships can automatically obtain control strategies according to the status of themselves and adjacent ships during operation, thereby achieving the purpose.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent collaborative technology, and in particular to a ship collaborative control method based on self-learning. Background Art

[0002] Since the 1980s, inspired by the study of biology and human sociology, multi-agent coordination control has become a research hotspot in many fields. Similar to the group advantages in biological and human societies such as fish hunting and bird migration, the coordination and cooperation between multiple individuals will greatly improve the intelligence of individual behavior, better complete many tasks that a single individual cannot complete, and have the advantages of high efficiency, high fault tolerance and inherent parallelism. To date, multi-agent coordination control technology has been widely used in wireless sensor networks, multi-robot collaborative assembly, drones, satellites, cluster spacecraft deep space exploration and other fields.

[0003] With the continuous development of marine resources, we have gradually leaped from shallow waters to deep waters, and put forward higher requirements for the efficiency of marine operations and the degree of automation of marine resource development. As a result, the coordination of multiple ships in civil and military operations has become more and more widely used. Compared with the operation of a single ship, the coordinated operation of multiple ships has many potential advantages: it can improve the efficiency of executing a certain operation; complete more complex tasks at sea, which may be impossible for a single ship to complete; improve the performance of the operation; and avoid the various effects of the operator's fatigue caused by the high difficulty of the operation and long-term operation. In addition, the coordinated operation of multiple ships also has the advantages of strong fault tolerance and strong adaptability. Therefore, the study of ship coordination has great practical application value, and the research on ship coordinated control has attracted more and more attention from scholars.

[0004] In the field of collaborative control, researchers often use methods such as leader-follower method, virtual pilot method and behavior method. Each method has its own related control method. Among them, leader-follower method is favored by researchers because of its good fault tolerance and low computing requirements. The multi-ship in this paper also adopts leader-follower method. In leader-follower method, one or more intelligent agents are set as leaders of the whole system. Usually, leaders do not need to receive messages from followers. It can be considered that leaders communicate directly with the main control center, and others are followers. Followers need to respond to instructions from leaders and change their own working status through information obtained from leaders to achieve their goals.

[0005] In the coordinated control of ships, how to enable ships to coordinate their own behaviors to be consistent with other ships is a key issue. At the same time, the input of the multi-agent system is generally constrained during operation, otherwise the system will become unstable. In response to the above-mentioned problems, the present invention proposes an input-constrained ship collaborative control method based on a self-learning method. Summary of the invention

[0006] The present invention mainly solves the technical problems existing in the above-mentioned prior art and provides a ship cooperative control method based on self-learning.

[0007] In order to achieve the above object, the present invention adopts the following technical solution, a ship cooperative control strategy optimization method based on self-learning, comprising the following steps:

[0008] Step S10, using the input module to obtain the parameters of each ship in the multi-agent system, including the expected running trajectory of the ship, and then mathematically modeling each ship in the multi-agent system;

[0009] Step S20, using the parameter and network initialization module, the network weight assignment module and the global loop module, the global loop is started, and the weight of the evaluation network is initialized by superimposing random assignment method on the current local optimal control strategy and the local optimal performance index;

[0010] Step S30, using the network training module and the local loop module, start the local loop, train the evaluation network after weight assignment through the adaptive dynamic programming method based on the initialized related parameters, iterate the update of the network weight in the direction of the gradient descent of the network minimization error function until the preset number of local loops is reached, obtain the iterative control law, and obtain the local optimal control strategy and the local optimal performance indicator function through the local loop;

[0011] Step S40, replacing the current iterative control law and performance index function with the iterative control law and the local optimal performance index function, and repeatedly executing steps S20 to S40 until the network meets the convergence accuracy, then the control strategy optimization of the ship is completed, and the ship collaborative control strategy is obtained by using the output module, and the optimization of the control strategy of the next ship is started;

[0012] The network minimization error function is:

[0013]

[0014] Among them, E i (t) Evaluate the network to minimize the error function;

[0015] The weight updating method of the evaluation network is as follows:

[0016] in

[0017] The local optimal performance index function is:

[0018]

[0019] Among them, Q ii ,R ij is a positive definite matrix; N i is the adjacent ship of the ith ship, add the vessel itself to the adjacent vessels; is a monotonically increasing odd function; φ(δ i ) is the activation function in the neural network; W i is the weight; is δ i The state equation of the ith ship is: The mathematical model of the leader is:

[0020] Preferably, the network convergence accuracy is 1.0×10 -5 .

[0021] Preferably, in step S20, the weights of the evaluation network are initialized by superimposing random assignments on the current local optimal control strategy and the local optimal performance index, and the method is:

[0022] Within the set range (-0.1, 0.1), the weights of the action network and the evaluation network are initialized by random assignment.

[0023] Preferably, the input module is configured to obtain relevant parameters of the ship cooperative control strategy and input them, and to construct an evaluation network based on a neural network;

[0024] The parameter and network initialization module is configured to initialize the relevant parameters and the network structure and set the network convergence accuracy;

[0025] The network weight assignment module is configured to initialize the weight of the evaluation network by randomly assigning values ​​within a set range;

[0026] The network training module is configured to train the evaluation network after weight assignment by an adaptive dynamic programming method based on the initialized relevant parameters, and update the network weights in the direction of gradient descent of the network minimization error function;

[0027] The local loop module is configured to iteratively update the network weights through the network training module until a preset number of local loops is reached, obtain a local optimal control strategy and a local optimal performance indicator function, and jump to the global loop module;

[0028] The global loop module is configured to replace the current iterative control law and the performance index function with the iterative control law and the local optimal performance index function, and repeatedly perform network weight assignment and network training until the network meets the convergence accuracy, thereby obtaining the optimal control strategy;

[0029] The output module is configured to output the acquired ship cooperative control strategy.

[0030] Beneficial Effects

[0031] The present invention provides a ship cooperative control method based on self-learning. It has the following beneficial effects:

[0032] (1) The self-learning-based ship collaborative control method combines the key technologies of self-learning optimization ADP algorithm and multi-agent system communication. With the support of these technologies, multiple ships can automatically obtain control strategies according to the status of themselves and adjacent ships during operation, thereby achieving the purpose.

[0033] (2) The ship cooperative control method based on self-learning comprehensively considers the weight convergence speed and stability. Within the global loop, the weight of the evaluation network is obtained by a random assignment method within the set range (-0.1, 0.1); the weight of the evaluation network is initialized within the global loop, and the weight can be re-initialized at the beginning of each global loop, thereby better realizing the convergence of the neural network on the basis of ensuring network stability and convergence speed, so as to find the optimal control strategy as soon as possible.

[0034] (3) The ship cooperative control method based on self-learning, according to the set number of iterations of the local loop, if the local loop has not ended at this time, continues to return to the beginning of step S30 to continue evaluating the network training; if the local loop has ended, continues to determine whether the current global loop has ended; if the global loop has not ended, returns to step S20 to re-initialize the ship cooperative control strategy optimization method based on self-learning, and then optimizes and adjusts the strategy according to the initialization result; if the global loop has ended, the optimal ship cooperative control strategy is obtained.

[0035] (4) In this ship cooperative control method based on self-learning, when the i-th agent can obtain information from the leader, c i >0, otherwise c i= 0. The output of the neural network is the approximate value function. The number of neurons in the input layer of the evaluation network is determined according to the number of states of the ship. The number of neurons in the hidden layer can be set by yourself, and the activation function can select different activation functions according to different ships in the multi-agent. The number of neurons in the output layer is 1, and the output is the approximate value function.

[0036] (5) The ship cooperative control method based on self-learning, through an input module, a parameter and network initialization module, a network weight assignment module, a network training module, a local loop module, a global loop module, and an output module, in order to obtain the relevant parameters of the ship cooperative control strategy and input and construct an evaluation network based on a neural network; initialize the relevant parameters and initialize the network structure and set the network convergence accuracy; initialize the weight of the evaluation network by a random assignment method within a set range; based on the initialized relevant parameters, train the evaluation network after weight assignment by an adaptive dynamic programming method, and update the network weight in the direction of gradient descent of the network minimization error function; iterate the network weight update by the network training module until a preset number of local loops is reached, obtain the local optimal control strategy and the local optimal performance indicator function, and jump to the global loop module; replace the current iterative control law and performance indicator function with the iterative control law and the local optimal performance indicator function, and repeat the network weight assignment and network training until the network meets the convergence accuracy, and obtain the optimal control strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for the implementation or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0038] Figure 1 It is a flow chart of the input-constrained ship cooperative control method based on self-learning of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment: A ship cooperative control method based on self-learning, such as Figure 1 As shown, the following steps are included:

[0041] Step S10, using the input module to obtain the parameters of each ship in the multi-agent system, including the expected trajectory of the ship, and then mathematically modeling each ship in the multi-agent system; for an entity that is easy to model, such as a ship, it is possible to avoid using a neural network to build a model, which will make the obtained control strategy more accurate. For the control of multi-agents, it is also necessary to know the expected trajectory so that the leader can give orders; at the beginning of the algorithm, the method of the present invention needs to initialize the global loop and local loop times and the network structure; pre-set the global loop and local loop times, and set the network convergence accuracy to 1.0×10 -5 .

[0042] In one embodiment of the present invention, the initialized network structure is a three-layer neural network, and the input of the neural network is δ i ,in x i is the state of the i-th ship, x0 is the state of the leader; a ij is the element value corresponding to the adjacency matrix, c i is a fixed gain. When the i-th agent can obtain messages from the leader, c i >0, otherwise c i = 0. The output of the neural network is the approximate value function. The number of neurons in the input layer of the evaluation network is determined according to the number of states of the ship. The number of neurons in the hidden layer can be set by yourself, and the activation function can select different activation functions according to different ships in the multi-agent. The number of neurons in the output layer is 1, and the output is the approximate value function.

[0043] Step S20, using the parameter and network initialization module, the network weight assignment module and the global loop module, the global loop is started, and the weight of the evaluation network is initialized by a random assignment method; according to the ship cooperative control strategy optimization method based on self-learning, the weight of the evaluation network needs to be initialized. In one embodiment of the present invention, the weight of the evaluation network is obtained by a random assignment method within a set range (-0.1, 0.1) within the global loop, taking into account the weight convergence speed and stability; the weight of the evaluation network is initialized within the global loop, and the weight can be re-initialized at the beginning of each global loop, so as to better achieve the convergence of the neural network on the basis of ensuring the network stability and convergence speed, so as to find the optimal control strategy as soon as possible;

[0044] Step S30, using the network training module and the local loop module, start the local loop, based on the initialized related parameters, train the evaluation network after weight assignment through an adaptive dynamic programming method, iteratively update the network weights in the direction of gradient descent of the network minimization error function until the preset number of local loops is reached, obtain the iterative control law, and obtain the local optimal control strategy and the local optimal performance indicator function through the evaluation network after the local loop.

[0045] User sets the start time t start and end time t close , within the set time period, firstly, under the given initial state and initial strategy, the network weight is updated using the following formula:

[0046] in

[0047] Where Q ii ,R ij is a positive definite matrix; N i is the adjacent ship of the ith ship, add the vessel itself to the adjacent vessels; is a monotonically increasing odd function; φ(δ i ) is the activation function in the neural network; W i is the weight; is δ i The state equation of the ith ship is: The mathematical model of the leader is:

[0048] Step S40, replacing the current iterative control law and performance index function with the iterative control law and the local optimal performance index function, that is:

[0049] The resulting value function is:

[0050] Steps S20 to S40 are repeatedly executed until the network meets the convergence accuracy, and the optimal ship cooperative control strategy is obtained.

[0051] According to the set number of iterations of the local loop, if the local loop has not ended at this time, continue to return to the beginning of step S30 to continue evaluating the network training. If the local loop has ended, continue to determine whether the current global loop has ended. If the global loop has not ended, return to step S20 to re-initialize the self-learning-based ship collaborative control strategy optimization method, and then optimize and adjust the strategy according to the initialization result; if the global loop has ended, the optimal ship collaborative control strategy is obtained, and the ship collaborative control strategy is obtained using the output module.

[0052] The input module is configured to obtain relevant parameters of the ship cooperative control strategy and input them and to construct an evaluation network based on the neural network;

[0053] The parameter and network initialization module is configured to initialize the relevant parameters and the network structure and set the network convergence accuracy;

[0054] The network weight assignment module is configured to initialize the weight of the evaluation network by randomly assigning values ​​within a set range;

[0055] The network training module is configured to train the evaluation network after weight assignment by an adaptive dynamic programming method based on the initialized relevant parameters, and update the network weights in the direction of gradient descent of the network minimization error function;

[0056] The local loop module is configured to iteratively update the network weights through the network training module until a preset number of local loops is reached, obtain a local optimal control strategy and a local optimal performance indicator function, and jump to the global loop module;

[0057] The global loop module is configured to replace the current iterative control law and the performance index function with the iterative control law and the local optimal performance index function, and repeatedly perform network weight assignment and network training until the network meets the convergence accuracy, thereby obtaining the optimal control strategy;

[0058] The output module is configured to output the acquired ship cooperative control strategy.

[0059] Working principle of the present invention:

[0060] In the present invention, the key technologies of self-learning optimization ADP algorithm and multi-agent system communication are combined. With the support of these technologies, multiple ships can automatically obtain control strategies according to the status of themselves and adjacent ships during operation, thereby achieving the purpose.

[0061] In the present invention, taking into account the convergence speed and stability of the weights, within the global loop, the weights of the evaluation network are obtained by a random assignment method within a set range (-0.1, 0.1); the weights of the evaluation network are initialized within the global loop, and the weights can be reinitialized at the beginning of each global loop, thereby better achieving the convergence of the neural network on the basis of ensuring the stability and convergence speed of the network, so as to find the optimal control strategy as soon as possible.

[0062] In the present invention, according to the set number of iterations of the local loop, if the local loop has not ended at this time, continue to return to the beginning of step S30 to continue to evaluate the network training; if the local loop has ended, continue to judge whether the current global loop has ended; if the global loop has not ended, return to step S20 to re-initialize the self-learning-based ship collaborative control strategy optimization method, and then optimize and adjust the strategy according to the initialization result; if the global loop has ended, the optimal ship collaborative control strategy is obtained.

[0063] In the present invention, when the i-th agent can obtain a message from the leader, c i >0, otherwise c i = 0. The output of the neural network is the approximate value function. The number of neurons in the input layer of the evaluation network is determined according to the number of states of the ship. The number of neurons in the hidden layer can be set by yourself, and the activation function can select different activation functions according to different ships in the multi-agent. The number of neurons in the output layer is 1, and the output is the approximate value function.

[0064] In the present invention, through an input module, a parameter and network initialization module, a network weight assignment module, a network training module, a local loop module, a global loop module, and an output module, in order to obtain the relevant parameters of the ship collaborative control strategy and input and construct an evaluation network based on a neural network; the relevant parameters are initialized, the network structure is initialized, and the network convergence accuracy is set; the weight of the evaluation network is initialized by a random assignment method within a set range; based on the initialized relevant parameters, the evaluation network after weight assignment is trained by an adaptive dynamic programming method, and the network weight is updated in the direction of gradient descent of the network minimization error function; the network weight is iteratively updated by the network training module until a preset number of local cycles is reached, a local optimal control strategy and a local optimal performance index function are obtained, and the global loop module is jumped; the current iterative control law and performance index function are replaced by the iterative control law and the local optimal performance index function, and the network weight assignment and network training are repeatedly performed until the network meets the convergence accuracy to obtain the optimal control strategy.

[0065] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A ship cooperative control strategy optimization method based on self-learning, characterized by: The following steps are involved: Step S10, using the input module to obtain the parameters of each ship in the multi-agent system, including the expected running trajectory of the ship, and then mathematically modeling each ship in the multi-agent system; Step S20, using the parameter and network initialization module, the network weight assignment module and the global loop module, the global loop is started, and the weight of the evaluation network is initialized by superimposing random assignment method on the current local optimal control strategy and the local optimal performance index; Step S30, using the network training module and the local loop module, start the local loop, train the evaluation network after weight assignment through the adaptive dynamic programming method based on the initialized related parameters, iterate the update of the network weight in the direction of the gradient descent of the network minimization error function until the preset number of local loops is reached, obtain the iterative control law, and obtain the local optimal control strategy and the local optimal performance indicator function through the local loop; Step S40, replacing the current iterative control law and performance index function with the iterative control law and the local optimal performance index function, and repeatedly executing steps S20 to S40 until the network meets the convergence accuracy, then the control strategy optimization of the ship is completed, and the ship collaborative control strategy is obtained by using the output module, and the optimization of the control strategy of the next ship is started; The network minimization error function is: Among them, E i (t) Evaluate the network to minimize the error function; The weight updating method of the evaluation network is as follows: in The local optimal performance index function is: Among them, Q ii ,R ij is a positive definite matrix; N i is the adjacent ship of the ith ship, add the vessel itself to the adjacent vessels; is a monotonically increasing odd function; φ(δ i ) is the activation function in the neural network; W i is the weight; is δ i The state equation of the ith ship is: The mathematical model of the leader is:

2. The ship cooperative control strategy optimization method based on self-learning according to claim 1 is characterized in that: The network convergence accuracy is 1.0×10 -5 .

3. The ship cooperative control strategy optimization method based on self-learning according to claim 1 is characterized in that: In step S20, the weights of the evaluation network are initialized by superimposing random assignments on the current local optimal control strategy and the local optimal performance index, and the method is: Within the set range (-0.1, 0.1), the weights of the action network and the evaluation network are initialized by random assignment.

4. The ship cooperative control strategy optimization method based on self-learning according to claim 1 is characterized in that: The input module is configured to obtain relevant parameters of the ship cooperative control strategy and input them and to construct an evaluation network based on the neural network; The parameter and network initialization module is configured to initialize the relevant parameters and the network structure and set the network convergence accuracy; The network weight assignment module is configured to initialize the weight of the evaluation network by randomly assigning values ​​within a set range; The network training module is configured to train the evaluation network after weight assignment by an adaptive dynamic programming method based on the initialized relevant parameters, and update the network weights in the direction of gradient descent of the network minimization error function; The local loop module is configured to iteratively update the network weights through the network training module until a preset number of local loops is reached, obtain a local optimal control strategy and a local optimal performance indicator function, and jump to the global loop module; The global loop module is configured to replace the current iterative control law and the performance index function with the iterative control law and the local optimal performance index function, and repeatedly perform network weight assignment and network training until the network meets the convergence accuracy, thereby obtaining the optimal control strategy; The output module is configured to output the acquired ship cooperative control strategy.

Images