Intelligent scheduling method, device and equipment and storage medium

Abstract

The invention relates to an intelligent scheduling method. The method comprises the steps of obtaining state information of a current environment; determining a compensation parameter corresponding to the current environment according to the state information; inputting the compensation parameter into a preset training model, and outputting a corresponding compensation action; executing the compensation action to realize intelligent scheduling; the advantages of a DQN algorithm, an experience playback buffer area and a target network are integrated, and a depth determination policy gradient (DDPG) algorithm is proposed to solve the deep reinforcement learning problem in a continuous state behavior space. However, due to the interaction between an Actor network and a Q function in the DDPG algorithm, the algorithm is generally difficult to reach stability, so that the DDPG algorithm is difficult to be directly applied to a complex high-dimensional multi-agent environment.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an intelligent scheduling method, device, equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, concepts such as Cyberspace and system movement confrontation have been continuously proposed, and related confrontation activities have developed into system confrontation supported by information technology. At the same time, the deepening and development of artificial intelligence technology research has accelerated the speed of using intelligent technology for command and control. With the rapid development of artificial intelligence, command and control systems are also developing in the direction of intelligence, and the emergence of deep learning has provided new ideas for the research of command and control systems, which is of great strategic significance.

[0003] However, the multi-dimensional motion space makes the motion environment and scenes more complex. The large amount of confrontation information makes the traditional mode of relying solely on manual analysis to make judgments no longer efficient. How to accurately grasp the battlefield situation and make accurate decisions has become a key issue in command and control. The intelligence of command and control systems has become a trend.

[0004] Therefore, there is an urgent need to develop an intelligent scheduling method, device, equipment and storage medium to solve one or more of the above-mentioned problems. Summary of the invention

[0005] In view of this, in order to solve the above-mentioned technical problems or part of the technical problems, this embodiment provides an intelligent scheduling method, device, equipment and storage medium.

[0006] In a first aspect, the present application provides an intelligent scheduling method, the method comprising:

[0007] Get the status information of the current environment;

[0008] Determine a compensation parameter corresponding to the current environment according to the state information;

[0009] Input the compensation parameters into a preset training model and output corresponding compensation actions;

[0010] The compensation action is performed to achieve intelligent scheduling.

[0011] In a possible implementation manner, determining the compensation parameter corresponding to the current environment according to the state information includes:

[0012] Analyze the state information to obtain noise parameters and excitation parameters of the current environment;

[0013] A compensation parameter corresponding to the current environment is determined based on the noise parameter and the excitation parameter.

[0014] In a possible implementation manner, the noise parameter is a first noise parameter or a second noise parameter;

[0015] The state information is analyzed to obtain noise parameters of the current environment, including:

[0016] Analyze the state information to obtain a first noise parameter and a second noise parameter;

[0017] Determine the time limit in the time domain under the current environment;

[0018] When the limit time is the first time, using the first noise parameter as the noise parameter of the current environment;

[0019] When the limit time is the second time, the second noise parameter is used as the noise parameter of the current environment.

[0020] In a possible implementation manner, the calculation formula of the first noise parameter is as follows:

[0021]

[0022] Among them, OU noise-k is the first noise parameter at time K, x n is the behavior parameter of the previous action, x n+1 is the next action behavior parameter, k is the time parameter, v is the forward speed, and -v is the reverse speed.

[0023] In a possible implementation, analyzing the state information to obtain the excitation parameters of the current environment includes:

[0024] Determine the state type of the current environment based on the state information;

[0025] According to the state type, the corresponding excitation parameter calculation rule is called, and the excitation parameter calculation rule is used to determine the excitation parameter of the current environment.

[0026] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is:

[0027] r t =-((x-200) 2 +(y-50) 2 ) 0.5 -(x-100);

[0028] Among them, r t is the excitation parameter, and x and y are the position coordinates.

[0029] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is:

[0030] r t =180-((x+sinr×40-50) 2 +(y+sinr×40-100) 2 ) 0.5 ;

[0031] Among them, r t is the excitation parameter, x and y are the position coordinates, and r is the radian value of the direction.

[0032] In a second aspect, the present application provides an intelligent scheduling device, the device comprising:

[0033] The acquisition module is used to obtain the status information of the current environment;

[0034] A determination module, used to determine compensation parameters corresponding to the current environment according to the state information;

[0035] A compensation module, used to input the compensation parameters into a preset training model and output corresponding compensation actions;

[0036] The execution module is used to execute the compensation action to achieve intelligent scheduling.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the intelligent scheduling method described in any one of the embodiments of the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the intelligent scheduling method described in any embodiment of the first aspect are implemented.

[0039] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: The method provided by the embodiment of the present application combines the advantages of the DQN algorithm, the experience replay buffer and the target network, and proposes a deep deterministic policy gradient (DDPG) algorithm to solve the deep reinforcement learning problem in the continuous state behavior space. However, due to the interaction between the Actor network and the Q function in the DDPG algorithm, it is usually difficult for the algorithm to achieve stability, so it is difficult to directly apply the DDPG algorithm to complex high-dimensional multi-agent environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present embodiments and, together with the description, serve to explain the principles of the present embodiments.

[0041] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0043] Figure 1 A flowchart of an intelligent scheduling method provided in an embodiment of the present application;

[0044] Figure 2 SD-DDPG framework diagram provided for the embodiment of the present application;

[0045] Figure 3 A comparison diagram of OU and Gaussian noise provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a stability comparison test of the SD-DDPG algorithm provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a convergence comparison test of the SD-DDPG algorithm provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of an intelligent scheduling device provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The disclosure below provides many different embodiments or examples to implement the different structures of the present embodiment. In order to simplify the disclosure of the present embodiment, the components and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present embodiment. In addition, the present embodiment can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0052] From AlphaGo's victory over Lee Sedol to the Real-Time Strategy (RTS) game AI, deep learning algorithms have made significant progress in the field of game systems. At the same time, a wave of research on deep learning in command and control systems has been set off. Deep reinforcement learning, especially in solving sequential decision-making problems, has solved many major problems. It combines artificial experience learning with decision-making thinking. The combination of the two is the development direction of modern adversarial sports decision-making. Reinforcement learning has the advantages of strong robustness, independence from environmental models and prior knowledge, and trial and error is often used to seek the optimal adversarial decision sequence when applied to adversarial actions.

[0053] Q-Learning is a typical reinforcement learning method. Q-Learning is widely used in some links of adversarial decision-making. The DeepMind team proposed the DQN algorithm, which combines deep convolutional neural networks and Q-learning. It has achieved the decision-making and control level of human experts on the Atari series of games and avoided the huge storage space of the Q table. In addition, the experience replay memory and target network are used to improve the stability of the training process. Later, a university proposed a technical framework for tactical decision-making of army units based on DQN-based inverse reinforcement learning based on the algorithm, which achieved certain results in solving tactical action decisions. However, although the DQN algorithm has achieved a series of results in discrete behavior decision-making, it is difficult to achieve high-dimensional continuous actions. If the continuously changing actions are infinitely divided, the number of actions will increase exponentially with the increase in degrees of freedom, which will lead to the problem of dimensional mutation and the network will be difficult to converge. Combining the advantages of the DQN algorithm, the experience replay buffer and the target network, the deep deterministic policy gradient (DDPG) algorithm is proposed to solve the deep reinforcement learning problem in the continuous state behavior space. However, due to the interaction between the Actor network and the Q function in the DDPG algorithm, the algorithm is usually difficult to achieve stability, so it is difficult to directly apply the DDPG algorithm to complex high-dimensional multi-agent environments.

[0054] In order to solve the problem that deep reinforcement learning algorithms are difficult to apply to modern battlefield environments with highly complex and continuous decision-making, and the algorithm is difficult to converge in a multi-agent environment, this embodiment proposes an improved DDPG algorithm - Single-mode and Double-noise DDPG algorithm (SD-DDPG), which improves the DDPG algorithm in terms of experience sampling, incentive function, exploration strategy and multi-agent framework.

[0055] Figure 1 A flow chart of an intelligent scheduling method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method specifically includes:

[0056] S101, obtaining status information of the current environment;

[0057] In this embodiment, first, this embodiment needs to obtain the state information of the current environment, which includes but is not limited to the temperature, humidity, light intensity and other relevant environmental parameters of the environment. This information will provide a comprehensive environmental overview for this embodiment, so that this embodiment can better understand and deal with the current environmental conditions.

[0058] S102, determining a compensation parameter corresponding to the current environment according to the state information;

[0059] In this embodiment, according to the state information obtained by this embodiment, this embodiment needs to determine the compensation parameters corresponding to the current environment. These compensation parameters are determined according to pre-set standards and rules, and are intended to adjust and compensate for adverse factors in the environment. For example, if the ambient temperature is too high, this embodiment may need to reduce the temperature compensation parameters to ensure that the device can operate at a suitable temperature.

[0060] S103, inputting the compensation parameters into a preset training model, and outputting corresponding compensation actions;

[0061] In this embodiment, the determined compensation parameters are input into a preset training model. This training model is trained and optimized with a large amount of data, and can output the most appropriate compensation action according to the input compensation parameters. These compensation actions may be to adjust the operating parameters of the device, change the working mode of the device, or other measures that can effectively cope with the current environmental conditions.

[0062] S104: Execute the compensation action to achieve intelligent scheduling.

[0063] In this embodiment, the compensation action of the model output needs to be executed to realize intelligent scheduling. In this way, the embodiment can ensure that the device can operate stably under various complex environmental conditions and improve the reliability and efficiency of the system. The realization of intelligent scheduling can not only improve the performance of the device, but also reduce maintenance costs and bring better user experience.

[0064] The intelligent scheduling method provided in this embodiment uses a continuous action space in the traditional DDPG algorithm. The action space that needs to be taken in a task round is very large. The discrete incentive function can only give the same incentive value within a certain action range, and it is impossible to accurately and effectively evaluate the subtle changes in the action, which makes it difficult for the model to converge. Based on this, the traditional DDPG algorithm is improved accordingly to address this problem, and the improved mixed noise is introduced into the DDPG algorithm to improve the exploration ability of the action space. By introducing mixed noise, the model can perform more detailed exploration in the action space, thereby more accurately evaluating the subtle changes in the action. This improvement enables the model to converge faster during the learning process and improves the accuracy and efficiency of intelligent scheduling. In addition, this embodiment also optimizes the design of the reward function so that it can better reflect the task objectives and environmental conditions, further improving the performance of intelligent scheduling. By implementing the improved DDPG algorithm, the intelligent scheduling device can schedule the equipment more intelligently to ensure stable operation in various complex environments.

[0065] In an optional solution of this embodiment, determining the compensation parameter corresponding to the current environment according to the state information includes:

[0066] Analyze the state information to obtain noise parameters and excitation parameters of the current environment;

[0067] A compensation parameter corresponding to the current environment is determined based on the noise parameter and the excitation parameter.

[0068] In this embodiment, by analyzing the described state information in detail, this embodiment can accurately obtain the noise parameters and excitation parameters in the current environment; on this basis, in combination with these noise parameters and excitation parameters, this embodiment can further determine the compensation parameters corresponding to the current environment.

[0069] This provides more accurate decision support for the intelligent dispatching device. The determination of compensation parameters is a key step for the intelligent dispatching device to adapt to environmental changes and optimize the dispatching strategy. In this way, the dispatching strategy can be dynamically adjusted according to real-time environmental changes to ensure the efficiency and stability of the system operation. In addition, the intelligent dispatching also integrates advanced data processing, which can process data from various sensors in real time and feed the processing results back to the dispatching algorithm to achieve rapid response and precise control.

[0070] In an optional solution of this embodiment, the noise parameter is a first noise parameter or a second noise parameter;

[0071] The state information is analyzed to obtain noise parameters of the current environment, including:

[0072] Analyze the state information to obtain a first noise parameter and a second noise parameter;

[0073] Determine the time limit in the time domain under the current environment;

[0074] When the limit time is the first time, using the first noise parameter as the noise parameter of the current environment;

[0075] When the limit time is the second time, the second noise parameter is used as the noise parameter of the current environment.

[0076] In this embodiment, by deeply analyzing and processing the above state information, this embodiment can successfully obtain the noise parameters in the current environment. This process includes the following key steps:

[0077] First, this embodiment needs to perform a detailed analysis on the described state information in order to extract the noise parameters in the current environment. This step is crucial because it provides basic data for evaluating and processing noise for this embodiment.

[0078] Next, this embodiment will focus on determining the noise parameters in the current environment, specifically, including two main noise parameters: a first noise parameter and a second noise parameter. These two parameters represent different aspects of noise characteristics, providing more comprehensive noise information for this embodiment.

[0079] In addition, this embodiment also needs to determine the limit time in the time domain under the current environment. This limit time refers to the time point at which the noise parameter may reach the maximum or minimum value under the current environmental conditions. Understanding this limit time is of great significance for predicting and controlling noise.

[0080] After determining the limit time, this embodiment will adopt different strategies according to the specific situation. Specifically, if the limit time is the first time, this embodiment will select the first noise parameter as the noise parameter of the current environment. This selection is based on the evaluation of the current environmental noise characteristics, ensuring that this embodiment can accurately reflect the noise conditions at the first time point.

[0081] On the contrary, if the limit time is the second time, the present embodiment will select the second noise parameter as the noise parameter of the current environment. This selection is also based on the evaluation of the current environmental noise characteristics, ensuring that the present embodiment can accurately reflect the noise condition at the second time point.

[0082] Through the above steps, the noise parameters in the current environment can be comprehensively analyzed and determined, providing a solid foundation for subsequent noise control and management.

[0083] In an optional solution of this embodiment, the calculation formula of the first noise parameter is as follows:

[0084]

[0085] Among them, OU noise-k is the first noise parameter at time K, x n is the behavior parameter of the previous action, x n+1 is the next action behavior parameter, k is the time parameter, v is the forward speed, and -v is the reverse speed.

[0086] For example, the action strategy of adding noise in the DDPG algorithm is independent of the learning strategy, that is, DDPG is a deterministic strategy, and the exploration noise can be set by itself. The original DDPG algorithm uses OU (Ornstein-Uhlenbeck) noise. The OU process is a random process, and its differential form formula is as follows:

[0087] dx t = -θ(x t -μ)dt+σdW t

[0088] Among them, μ is the mean, θ indicates the speed at which the noise tends to the mean, and σ indicates the degree of noise fluctuation. OU noise is time-series-related exploration noise, that is, the noise of the previous step will affect the noise of the next step, and it is a Markov model. It is precisely based on the time-series-related characteristics of OU noise that the exploration efficiency of inertial systems will be higher. DDPG, as a representative of continuous algorithms, is suitable for inertial systems. Many reinforcement learning algorithms also often use Gaussian noise, taking the output action of the policy network in the reinforcement learning algorithm as the mean, and directly superimposing the Gaussian distribution ε~N(0,σ2) as the exploration strategy of the reinforcement learning algorithm. Different from the time-series correlation of OU noise, Gaussian noise will not be affected by the previous action, so it is very suitable for decision actions that are not time-series-related. In the confrontation process, the agent has three decision actions, namely: speed control, direction control, and shooting action. The control of speed and direction is applicable to inertial systems. The use of OU noise can improve the exploration efficiency of the combat agent in speed control and direction selection strategies. However, for the shooting action of the combat agent, since the execution of the shooting action is not correlated in time sequence, that is, the shooting action of the previous step will not affect whether the shooting action is taken in the next step, the use of OU noise will reduce the exploration efficiency of the shooting decision action. This embodiment introduces the mixed dual noise of OU+Gaussian to improve the DDPG algorithm, improve the exploration efficiency and convergence speed of the algorithm in a military simulation environment, and the specific implementation method is as follows:

[0089] The DDPG algorithm introduces improved OU noise analysis, the details are as follows:

[0090] The DDPG algorithm is used to predict the speed control and direction control of the behavior, because these control behaviors have certain correlation and timing. In the DDPG algorithm, the Ornstein-Uhlenbeck random process (hereinafter referred to as the OU process) is used as the introduced random noise: because the OU process has good correlation in timing. The mathematical expression is as follows:

[0091] A=r t +OU noise-k ;

[0092] A is the action prediction, r t is the improved DDPG algorithm excitation function in this embodiment, OU noise-k

[0093] is the K-order analysis function of OU noise, and its mathematical derivation is as follows:

[0094] The discrete form of the simple OU noise generation process is as follows:

[0095] x n+1 =x n +θ(μ-xn )Δt+σΔω n ;

[0096] x n is the previous action, x n+1 is the next action behavior, where μ represents the mean of the noise, θ controls the speed at which the noise returns to the mean, σ controls the size of the random disturbance, and ω n represents the Wiener process.

[0097] In this embodiment, the OU noise is further refined, x n+1 For the next action behavior, not only n The previous action is associated with the previous action, and considering the relevance of the K-th action, this embodiment establishes a K-order OU noise analysis method, and the mathematical expression is as follows:

[0098]

[0099] OU noise-k It is not only related to the noise of the previous step, but also to the noise of the previous X steps. Such noise enables the algorithm to better explore in an inertial environment. φ is the partial derivative of the position function at the positive velocity V, the negative velocity -V, and the K+1 time.

[0100] In this embodiment, by introducing K-order OU noise analysis, the noise characteristics in an inertial environment can be simulated more accurately. The K-order OU noise analysis method takes into account the correlation of the first K action behaviors, making the calculation of noise parameters more complicated, but at the same time more in line with the noise changes in the actual environment. Specifically, the K-order OU noise analysis method adjusts the noise parameters of the current action behavior by calculating the partial derivatives of the first K action behaviors, so that the algorithm can more accurately reflect the dynamic changes of the environment when predicting and controlling noise.

[0101] In addition, this embodiment also considers the introduction of Gaussian noise to make up for the shortcomings of OU noise in non-time-related decision-making actions. The introduction of Gaussian noise makes the algorithm more flexible and efficient when processing non-time-related decisions such as shooting actions. By combining OU noise with Gaussian noise, the intelligent scheduling device of this embodiment can better adapt to complex and changing environments and improve the overall scheduling efficiency and accuracy.

[0102] In practical applications, the intelligent scheduling device of this embodiment can dynamically adjust the noise parameters according to the real-time changes of the environmental noise, thereby achieving effective control of the environmental noise. For example, in a military simulation environment, the intelligent scheduling device can adjust the behavior strategy of the combat agent in real time to adapt to the changes in the battlefield environment and improve the survivability and combat efficiency of the combat agent. In the civilian field, such as in intelligent transportation systems, the device can also optimize the scheduling of traffic flow according to the changes in traffic noise, reduce traffic congestion, and improve road use efficiency.

[0103] OU noise analysis mixed Gaussian noise, the specific method is as follows:

[0104] First, observe the characteristics of OU noise and Gaussian noise, such as Figure 3 As shown in the figure, it can be seen that OU noise is actually the superposition of several Gaussian noises on the time axis. Therefore, the important difference between OU noise and Gaussian noise is that after an infinite time limit, OU noise does not decay to 0 like Gaussian noise, but the process converges to a certain Gaussian distribution. Therefore, this embodiment uses the limit time T in the time domain to distinguish the two types of noise. The mixed noise designed in this embodiment is H noise for:

[0105]

[0106] In mixed noise, when time T tends to infinity, if the mean μ is greater than 0, OU noise is mainly used; if the mean μ is equal to 0, Gaussian noise is mainly used. This mixed noise strategy combines the advantages of OU noise in time-related exploration and the flexibility of Gaussian noise in non-time-related decision-making. In a military simulation environment, the shooting action of the intelligent agent often requires fast and independent decision-making. Therefore, this embodiment mainly uses Gaussian noise in the shooting action to improve the randomness and flexibility of the decision. At the same time, for speed control and direction control, this embodiment mainly uses K-order OU noise to enhance the algorithm's exploration ability in an inertial environment. Through this mixed noise strategy, the intelligent scheduling device of this embodiment can more effectively adapt to the complex and changeable military simulation environment and improve the combat effectiveness of the intelligent agent.

[0107] In an optional solution of this embodiment, the state information is analyzed to obtain the excitation parameters of the current environment, including:

[0108] Determine the state type of the current environment based on the state information;

[0109] According to the state type, the corresponding excitation parameter calculation rule is called, and the excitation parameter calculation rule is used to determine the excitation parameter of the current environment.

[0110] In this embodiment, after analyzing and evaluating the specific state information of the current environment, the embodiment can clearly identify the state type of the current environment. This process involves comprehensive analysis of various data and information to ensure that the embodiment can accurately determine the current state of the environment. Once the embodiment determines the state type of the environment, the next step is to select and call the corresponding incentive parameter calculation rule according to this type.

[0111] These incentive parameter calculation rules are pre-set and are intended to provide appropriate incentive parameters according to different environmental state types. By applying these rules, this embodiment can calculate incentive parameters that match the current environmental state. These parameters will be used to guide and adjust various behaviors and decisions in the environment to ensure that the best operating effect and efficiency can be achieved under specific conditions.

[0112] Specifically, the incentive parameters may include various incentives, such as rewards, penalties, incentive mechanisms, etc., which are all intended to mobilize individuals or systems in the environment to exhibit optimal behaviors in a specific state. By accurately calculating and applying these incentive parameters, the present embodiment can effectively guide and motivate the various components in the environment to achieve predetermined goals and desired effects.

[0113] The calculation rules of incentive parameters are based on a series of preset algorithms and models that take into account the dynamic changes of the environment and the complexity of individual behavior. For example, when the environment is in a competitive state, the incentive parameter calculation rules may increase the intensity of rewards to encourage individuals to take more proactive behaviors. On the contrary, in a cooperative state, the incentive parameters may focus more on long-term rewards and rewards for teamwork to promote the maximization of collective interests.

[0114] In addition, the calculation of incentive parameters may also include an evaluation of the individual's historical behavior to ensure that the incentive measures reflect the individual's actual performance and contribution. In this way, the incentive parameters can not only promote the optimal behavior in the current environment, but also guide the long-term development and growth of the individual.

[0115] In practical applications, the calculation and application of incentive parameters require a high degree of flexibility and adaptability to cope with the ever-changing environment and individual behavior. Therefore, the intelligent scheduling device of this embodiment can adjust the incentive parameters in real time to ensure that it can maintain high efficiency and adaptability in various complex and changing environments. Through this dynamic adjustment mechanism, the intelligent scheduling device can provide continuous and targeted incentives for individuals in the environment, thereby achieving long-term and sustainable development goals.

[0116] In an optional solution of this embodiment, when the state type is the first state, the calculation formula of the excitation parameter is:

[0117] r t =-((x-200) 2 +(y-50) 2 ) 0.5 -(x-100);

[0118] Among them, r t is the excitation parameter, and x and y are the position coordinates.

[0119] In an optional solution of this embodiment, when the state type is the first state, the calculation formula of the excitation parameter is:

[0120] r t =180-((x+sinr×40-50) 2 +(y+sinr×40-100) 2 ) 0.5 ;

[0121] Among them, r t is the excitation parameter, x and y are the position coordinates, and r is the radian value of the direction.

[0122] For example, the traditional DDPG algorithm uses a continuous action space. The action space that needs to be taken in a task round is very large. The discrete incentive function can only give the same incentive value within a certain action range, and it is impossible to accurately and effectively evaluate the subtle changes in the action, making it difficult for the model to converge. This embodiment is aimed at the traditional DDPG algorithm and makes corresponding improvements to this problem. The specific method is as follows:

[0123] A continuous excitation function with continuous excitation support is designed. The excitation function formula is as follows:

[0124]

[0125] (x, y) is the position coordinate of the combat agent, and r is the radian value of the direction. When the combat agent crosses the boundary of the environment or collides with an obstacle, the incentive value is set to -300 and a penalty is given. When the adversarial agent performs a shooting action but misses the target, an incentive value of -150 is accumulated. In order to guide the combat agent to learn the optimal adversarial decision strategy faster, a continuity function is designed to guide the adversarial agent to reach a pre-established area. The smaller the distance value, the greater the incentive value. After arriving near the designated area, the distance between the shooting point and the target base is set as the incentive function to guide the combat agent to shoot at the opponent's target base. Continuous incentive stimulation can more efficiently guide the agent to quickly learn the optimal decision sequence. The ultimate goal of this combat mission is to destroy the opponent's target base, giving an incentive value of +150.

[0126] like Figure 2As shown, this embodiment can accurately predict speed control, direction control, and shooting actions in large-scale adversarial behaviors. Subsequent experimental results reveal that by adopting the DDPG algorithm improved by hybrid dual noise, we achieve faster convergence speed and higher stability. When the DDPG algorithm is directly applied to a multi-agent adversarial decision-making environment, the algorithm is often difficult to converge. Therefore, this embodiment proposes an enhanced multi-agent flexible framework, namely the SD-DDPG framework.

[0127] In the single training mode, the single adversarial agent specified by the algorithm interacts with the environment independently and learns decision-making strategies, while other combat agents are temporarily dormant and do not interfere with the environment. Once the single training mode is exited, multiple agents will explore the environment at the same time and learn multi-agent collaborative strategies. By introducing the multi-agent framework in single mode, the convergence stability and rate of the algorithm have been significantly improved, which not only ensures that multiple agents can learn collaborative strategies, but also gives each adversarial agent a certain degree of independence.

[0128] like Figure 4 As shown in the figure, under the same noise environment, the comparison was made through 300 rounds of iterative training, and the maximum number of training times for each round was set to 500,000. The experimental results show that when OU noise is superimposed on speed control and direction control, and Gaussian noise is superimposed on shooting action control, the SD-DDPG algorithm shows higher stability in this military decision-making simulation environment.

[0129] like Figure 5 As shown in the figure, in order to solve the problem of algorithm instability and difficulty in convergence caused by dynamic changes in the environment under the multi-agent framework, this embodiment introduces a single training mode. Experiments show that the convergence speed and stability of the SD-DDPG algorithm in the single mode are significantly improved.

[0130] Based on the improved method of this embodiment, the SD-DDPG algorithm performs better in solving the adversarial decision-making problem in terms of stability and convergence than the traditional algorithm. It can more effectively improve the efficiency of intelligent adversarial decision-making.

[0131] Figure 6 A schematic diagram of an intelligent scheduling device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device specifically includes:

[0132] The acquisition module 601 is used to obtain the status information of the current environment;

[0133] A determination module 602 is used to determine a compensation parameter corresponding to the current environment according to the state information;

[0134] The compensation module 603 is used to input the compensation parameters into a preset training model and output a corresponding compensation action;

[0135] The execution module 604 is used to execute the compensation action to achieve intelligent scheduling.

[0136] In a possible implementation manner, the determination module 602 is further configured to analyze the state information to obtain noise parameters and excitation parameters of the current environment; and determine compensation parameters corresponding to the current environment based on the noise parameters and the excitation parameters.

[0137] In one possible implementation, the determination module 602 is also used to analyze the state information to obtain a first noise parameter and a second noise parameter; determine a limit time in the time domain under the current environment; when the limit time is the first time, use the first noise parameter as the noise parameter of the current environment; when the limit time is the second time, use the second noise parameter as the noise parameter of the current environment.

[0138] In a possible implementation, the calculation formula of the first noise parameter is as follows:

[0139]

[0140] Among them, OU noise-k is the first noise parameter at time K, x n is the behavior parameter of the previous action, x n+1 is the next action behavior parameter, k is the time parameter, v is the forward speed, and -v is the reverse speed.

[0141] In a possible implementation, the determination module 602 is further configured to determine a state type of the current environment based on the state information; call a corresponding excitation parameter calculation rule according to the state type, and determine an excitation parameter of the current environment using the excitation parameter calculation rule.

[0142] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is: t =-((x-200) 2 +(y-50) 2 ) 0.5 -(x-100); where r t is the excitation parameter, and x and y are the position coordinates.

[0143] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is:

[0144] r t =180-((x+sinr×40-50) 2+(y+sinr×40-100) 2 ) 0.5 ;

[0145] Among them, r t is the excitation parameter, x and y are the position coordinates, and r is the radian value of the direction.

[0146] The intelligent scheduling device provided in this embodiment can be as follows Figure 6 The intelligent scheduling device shown in can execute the following Figure 1-5 All steps of intelligent scheduling in Figure 1-5 The technical effect of intelligent scheduling is shown in the figure. For details, please refer to Figure 1-5 For the sake of brevity, the relevant description is not repeated here.

[0147] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, an embodiment of the present application provides an electronic device, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704; the memory 703 is used to store computer programs; the processor 701 is used to execute the program stored in the memory 703, and implement the steps of the intelligent scheduling method provided by any of the aforementioned method embodiments:

[0149] Obtain status information of the current environment; determine compensation parameters corresponding to the current environment based on the status information; input the compensation parameters into a preset training model and output corresponding compensation actions; execute the compensation actions to achieve intelligent scheduling.

[0150] In a possible implementation, the state information is analyzed to obtain noise parameters and excitation parameters of the current environment; and compensation parameters corresponding to the current environment are determined based on the noise parameters and the excitation parameters.

[0151] In one possible implementation, analyzing the state information to obtain the noise parameters of the current environment includes: analyzing the state information to obtain a first noise parameter and a second noise parameter; determining a limit time in the time domain under the current environment; when the limit time is the first time, using the first noise parameter as the noise parameter of the current environment; when the limit time is the second time, using the second noise parameter as the noise parameter of the current environment.

[0152] In a possible implementation manner, the calculation formula of the first noise parameter is as follows:

[0153]

[0154] Among them, OU noise-k is the first noise parameter at time K, x n is the behavior parameter of the previous action, x n+1 is the next action behavior parameter, k is the time parameter, v is the forward speed, and -v is the reverse speed.

[0155] In a possible implementation, the state type of the current environment is determined based on the state information; according to the state type, a corresponding excitation parameter calculation rule is called, and the excitation parameter calculation rule is used to determine the excitation parameter of the current environment.

[0156] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is:

[0157] r t =-((x-200) 2 +(y-50) 2 ) 0.5 -(x-100);

[0158] Among them, r t is the excitation parameter, and x and y are the position coordinates.

[0159] In a possible implementation manner, when the state type is the first state, the calculation formula of the excitation parameter is:

[0160] r t =180-((x+sinr×40-50) 2 +(y+sinr×40-100) 2 ) 0.5 ;

[0161] Among them, r t is the excitation parameter, x and y are the position coordinates, and r is the radian value of the direction.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0163] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0164] The above description is only a specific implementation of the present embodiment, so that those skilled in the art can understand or implement the present embodiment. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present embodiment. Therefore, the present embodiment will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. An intelligent scheduling method, characterized in that: include: Get the status information of the current environment; Determine a compensation parameter corresponding to the current environment according to the state information; Input the compensation parameters into a preset training model and output corresponding compensation actions; The compensation action is performed to achieve intelligent scheduling.

2. The method according to claim 1, characterized in that The determining, according to the state information, a compensation parameter corresponding to the current environment includes: Analyze the state information to obtain noise parameters and excitation parameters of the current environment; A compensation parameter corresponding to the current environment is determined based on the noise parameter and the excitation parameter.

3. The method according to claim 2, characterized in that The noise parameter is a first noise parameter or a second noise parameter; The state information is analyzed to obtain noise parameters of the current environment, including: Analyze the state information to obtain a first noise parameter and a second noise parameter; Determine the time limit in the time domain under the current environment; When the limit time is the first time, using the first noise parameter as the noise parameter of the current environment; When the limit time is the second time, the second noise parameter is used as the noise parameter of the current environment.

4. The method according to claim 3, characterized in that The calculation formula of the first noise parameter is as follows: Among them, OU noise-k is the first noise parameter at time K, x n is the behavior parameter of the previous action, x n+1 is the next action behavior parameter, k is the time parameter, v is the forward speed, and -v is the reverse speed.

5. The method according to claim 2, characterized in that: The state information is analyzed to obtain the excitation parameters of the current environment, including: Determine the state type of the current environment based on the state information; According to the state type, the corresponding excitation parameter calculation rule is called, and the excitation parameter calculation rule is used to determine the excitation parameter of the current environment.

6. The method according to claim 5, characterized in that When the state type is the first state, the calculation formula of the excitation parameter is: r t =-((x-200) 2 +(y-50) 2 ) 0.5 -(x-100); Among them, r t is the excitation parameter, and x and y are the position coordinates.

7. The method according to claim 5, characterized in that When the state type is the first state, the calculation formula of the excitation parameter is: r t =180-((x+sinr×40-50) 2 +(y+sinr×40-100) 2 ) 0.5 ; Among them, r t is the excitation parameter, x and y are the position coordinates, and r is the radian value of the direction.

8. An intelligent scheduling device, characterized in that: include: The acquisition module is used to obtain the status information of the current environment; A determination module, used to determine compensation parameters corresponding to the current environment according to the state information; A compensation module, used to input the compensation parameters into a preset training model and output corresponding compensation actions; The execution module is used to execute the compensation action to achieve intelligent scheduling.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the intelligent scheduling method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent scheduling method described in any one of claims 1 to 7 are implemented.

Images