High-dimensional intelligent control model structure and training method
By introducing the cooperative state matrix ω into the intelligent control model, linking the global state neural network and multi-dimensional neural network, the problem of high computing resources and time consumption in high-dimensional output is solved, and efficient high-dimensional intelligent control is achieved.
Patent Information
- Application Number
- CN202411293323.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2025-02-14
AI Technical Summary
In intelligent control scenarios that require high-dimensional output, the existing technology faces the problem of high computing resources and time consumption, resulting in limited control effects.
A high-dimensional intelligent control model structure is designed, and by introducing a cooperative state matrix ω, linking the global state neural network and multi-dimensional neural network, optimizing the connection relationship between the hidden layer and the output layer, and improving computing efficiency.
This method effectively improves the computing efficiency in high-dimensional output, improves the reliability of intelligent control methods, and can achieve better control effects in high-dimensional control scenarios.
Smart Images
Figure CN119443191A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of application of neural network structure and training method to intelligent control technology, and in particular to a high-dimensional intelligent control model structure and training method. Background Art
[0002] As a method and model of machine learning, neural network usually consists of input layer, hidden layer and output layer. The hidden layer processes information between the input layer and the output layer. It has shown strong capabilities in tasks such as image recognition, speech recognition, natural language processing, and intelligent decision-making and control. Intelligent control refers to a control method that uses artificial intelligence technologies such as machine learning, deep learning, and reinforcement learning to enable the control system to autonomously perceive environmental information, make autonomous decisions, and control. Intelligent control aims to improve the adaptability, self-learning ability, and intelligence level of the control system to meet the needs of complex systems and dynamic environments.
[0003] At present, intelligent control methods have problems in efficiency and computational complexity when facing high-dimensional output requirements. Some intelligent control methods, such as DQN, MADDPG, etc., require a lot of computing resources and time for training and decision-making in scenarios where high-dimensional output is required. When the control dimension required to be output by a general neural network structure is high, the control effect will be affected and limited to a certain extent, generally due to the following two reasons: First, the parameter space increases: the number of parameters of the neural network is related to the network structure and the number of layers, and the increase in parameter space will increase the complexity of training and computational cost. When the output dimension is high, the network needs more parameters to represent and learn the complexity of the output, which may lead to overfitting and training difficulties. Second, data imbalance: when the output dimension is high, the data distribution in different dimensions may become unbalanced. This will cause the network to have low output prediction accuracy in some dimensions and high accuracy in other dimensions, thereby affecting the overall control effect. To solve the above problems, a high-dimensional intelligent control model structure and training method are developed. Summary of the invention
[0004] The purpose of the present invention is to provide a high-dimensional intelligent control model structure and training method to solve the problem of how to improve the computing efficiency of intelligent control high-dimensional output and further improve the reliability of the intelligent control method in scenarios where high-dimensional output is required, where a large amount of computing resources and time are required for training and decision-making.
[0005] In a first aspect, the present invention provides a high-dimensional intelligent control model structure, comprising:
[0006] The high-dimensional intelligent control model includes a hidden layer, an output layer, a cooperation state matrix ω configured to be hidden in the model structure, and a plurality of neurons arranged in a layer sequence;
[0007] The ω structure plays a role in linking the global state neural network part and the multi-dimensional neural network;
[0008] The hidden layers are set L 1 Layer, L 2 Layer and L 3 Layer, each hidden layer is set with neuron nodes;
[0009] The information of each control dimension is composed of status information. And the composition structure of the action space
[0010] Determine multiple control dimension information, according to the state information composition form in the control dimension information And the composition structure of the action space You can Expressed as Q value Right now
[0011] In a second aspect, a high-dimensional intelligent control model structure training method includes:
[0012] Acquire high-dimensional intelligent control model information, wherein each piece of high-dimensional intelligent control model information includes multiple control dimension information and feature information;
[0013] Set the parameters of the evaluation network and the target network;
[0014] Performing data analysis on the multiple control dimension information and feature information as well as the parameters of the evaluation network and the parameters of the target network to obtain a dual network output result, wherein the dual network output result includes an evaluation network output result and a target network output result;
[0015] According to the dual network output results, the error calculation is performed to obtain the dual network error result, and the dual network error result is back-transferred to update the evaluation network parameters;
[0016] The experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain training results.
[0017] Furthermore, the loss function of the evaluation network includes:
[0018] The loss function for evaluating the network is;
[0019] Loss(θ)=E[||Qtar x -Qevl x ||].
[0020] Qtar x ,Qevl xRepresent the following two sets of vectors;
[0021]
[0022] According to the Bellman equation, we can get;
[0023]
[0024] in Represents the actions taken in K dimensions.
[0025] Further, the experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain training results, including;
[0026] The experience data (s,a,r,s′) is stored, where (s,a,r,s′)s is the initial environment state information, a is the action taken by the agent, r is the reward value obtained by the action, and s′ is the environment state information after action a acts on the environment. Small batch samples are randomly sampled for each step of training to obtain training results.
[0027] The beneficial effects of the present invention are as follows: The present invention provides a high-dimensional intelligent control model structure and training method. The key point of the technical solution of this application lies in the design and use of the ω structure. While the ω structure plays the role of linking the global state neural network part and the multi-dimensional neural network part, it ensures the realization effect of high-dimensional control and upgrades the DQN method that can only be used for single control in the past to a high-dimensional control method. At the same time, the solution also designs a specific neural network training method for the model structure and gives a data flow process. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0029] Figure 1 A schematic diagram of a high-dimensional intelligent control model structure and training method flow chart provided in an embodiment of the present invention.
[0030] Figure 2 A schematic diagram of a high-dimensional intelligent control model structure and training method flow chart provided in an embodiment of the present invention.
[0031] Figure 3 A schematic diagram of a high-dimensional intelligent control model structure and training method flow chart provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail in conjunction with the drawings.
[0033] See also Figure 1 In a first aspect, the present invention provides a high-dimensional intelligent control model structure, comprising:
[0034] The high-dimensional intelligent control model includes a hidden layer, an output layer, a cooperation state matrix ω configured to be hidden in the model structure, and a plurality of neurons arranged in a layer sequence;
[0035] The ω structure plays a role in linking the global state neural network part and the multi-dimensional neural network;
[0036] The hidden layers are set L 1 Layer, L 2 Layer and L 3 Layer, each hidden layer is set with neuron nodes;
[0037] The information of each control dimension is composed of status information. And the composition structure of the action space
[0038] Determine multiple control dimension information, according to the state information composition form in the control dimension information And the composition structure of the action space You can Expressed as Q value Right now
[0039] In a second aspect, a high-dimensional intelligent control model structure training method includes:
[0040] In step S101, high-dimensional intelligent control model information is obtained, wherein each piece of high-dimensional intelligent control model information includes a plurality of control dimension information and feature information;
[0041] In step S102, parameters of the evaluation network and parameters of the target network are set;
[0042] In step S103, data analysis is performed on the multiple control dimension information and feature information as well as the parameters of the evaluation network and the parameters of the target network to obtain a dual network output result, wherein the dual network output result includes an evaluation network output result and a target network output result;
[0043] In step S104, according to the dual network output results, an error calculation is performed to obtain a dual network error result, and the dual network error result is reversely transferred to update the evaluation network parameters;
[0044] In step S105, the experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain a training result;
[0045] Specifically, the loss function of the evaluation network includes:
[0046] The loss function of the evaluation network is:
[0047] The loss function for evaluating the network is;
[0048] Loss(θ)=E[||Qtar x -Qevl x ||].
[0049] Qtar x ,Qevl x Represent the following two sets of vectors;
[0050]
[0051] According to the Bellman equation, we can get;
[0052]
[0053] in Represents the actions taken in K dimensions.
[0054] Specifically, the experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain training results, including;
[0055] The experience data (s,a,r,s′) is stored, where (s,a,r,s′)s is the initial environment state information, a is the action taken by the agent, r is the reward value obtained by the action, and s′ is the environment state information after action a acts on the environment. Small batch samples are randomly sampled for each step of training to obtain training results.
[0056] The technical solution of this application is based on the multi-dimensional action control neural network structure of DQN;
[0057] In this scheme, the connection relationship between the hidden layer and the output layer is adjusted. The adjustment structure is realized by a cooperative state matrix ω hidden in the structure. As shown in the figure below, ω realizes a function similar to that of a track. The part of the neural network before ω contains the global control method information, while the part after ω represents the independent control dimension information.
[0058] exist Figure 2 Medium 1 ,L 2 , and L 3 Represent three hidden layers, n1, n2, and n3 represent the number of neural nodes in each hidden layer, and m represents the size of the action space for DQN output for a single control dimension.
[0059] in Represents the information of each control dimension, according to the composition form of the status information And the composition structure of the action space You can Expressed as Q value .Right now
[0060] The structural training method in the technical solution of this application;
[0061] See also Figure 3 For the training of the neural network structure, we adopted a dual network structure similar to the traditional DQN. Assume that the parameters of the evaluation network are θ and the parameters of the target network are θ - .
[0062] The loss function for evaluating the network is
[0063]
[0064] Qtar x ,Qevl x Represent the following two sets of vectors
[0065]
[0066] According to the Bellman equation, we can get
[0067]
[0068] in represents the actions taken in K dimensions. In addition to the dual network structure, the empirical data (s, a, r, s′) can also be stored and randomly sampled in small batches for each step of training. This method of randomly sampling empirical data can further break the continuity between data and speed up the convergence of the model.
[0069] For the neural network structure of the scheme design, the specific implementation methods of the following two parts of the structure can be flexibly changed, and the changes of these two parts of the structure will not affect the effect of the neural network structure of the scheme design.
[0070] The first part is the global neural network structure before the ω layer. This solution adopts a double-layer fully connected design. The specific structure of this part is not the key point emphasized by this solution, and the solution effect can also be achieved by using other link structures.
[0071] The second part is the sub-neural network structure of multiple dimensions after the ω layer. This solution uses multiple identical double-layer fully connected structures as the neural network structure of each specific dimension. The specific structure of this part is not the key point emphasized in this solution. Users can target the neural network structure of the output layer according to specific needs to achieve the best effect.
[0072] The technical solution disclosed in this application is a high-dimensional intelligent control model structure and training method. The main key point of the technical solution of this application lies in the design and use of the ω structure. While the ω structure plays the role of linking the global state neural network part and the multi-dimensional neural network part, it ensures the implementation effect of high-dimensional control and upgrades the DQN method that can only be used for single control in the past to a high-dimensional control method. At the same time, the solution also designs a specific neural network training method for the model structure and gives the data flow process. It solves the problem of how to improve the computing efficiency of intelligent control during high-dimensional output when a large amount of computing resources and time are required for training and decision-making in scenarios where high-dimensional output is required, and further improve the reliability of intelligent control methods.
[0073] Regarding the relevant terms in the technical field of this application, common methods and basic concepts in the field of intelligent control:
[0074] Reinforcement Learning(RL):
[0075] Reinforcement Learning (RL) is a machine learning method that learns the optimal behavior strategy through the interaction between the agent and the environment. The agent adjusts its decision-making based on the feedback from the environment to maximize the cumulative reward. Reinforcement learning is widely used in robot control, game strategy, autonomous driving and other fields.
[0076] Q-learning:
[0077] Q-learning is a reinforcement learning algorithm based on value iteration. It learns the optimal action strategy by establishing a Q-value table. The Q-value represents the value obtained by taking a certain action in a given state. Q-learning enables the agent to make the best decision in the environment by continuously updating the Q-value. The algorithm optimizes the strategy through trial and error and feedback, and gradually converges to the optimal solution. Q-learning has achieved good results in simple environments and has been applied in many fields such as robot control and autonomous driving.
[0078] Deep Q-Network(DQN):
[0079] DQN (Deep Q-Network) is a reinforcement learning algorithm based on deep learning. It combines Q-learning and deep neural networks to solve reinforcement learning problems with high-dimensional state spaces. DQN takes the state as input, outputs the Q value of the corresponding action, and then selects the optimal action based on the Q value. By continuously learning and updating network parameters, DQN is able to gradually optimize strategies and achieve high-performance decision-making in complex environments. DQN has made important breakthroughs and applications in many fields, such as game play and robot control.
[0080] Multi-Agent Actor-Critic(MAAC):
[0081] MAAC (Multi-Agent Actor-Critic) is a multi-agent reinforcement learning algorithm. It achieves collaborative decision-making and learning among multiple agents by applying the Actor-Critic algorithm to the multi-agent system. Each agent has its own Actor network and Critic network. The Actor network is used to generate action strategies, and the Critic network is used to evaluate the value of the strategy. Through collaborative learning and shared experience, MAAC enables agents to achieve efficient decision-making and cooperation in complex multi-agent environments. The MAAC algorithm has achieved good performance and application in the fields of multi-agent collaborative tasks and multi-agent games.
[0082] Multi-Agent Deep Deterministic Policy Gradient(MADDPG):
[0083] MADDPG (Multi-Agent Deep Deterministic Policy Gradient) is a multi-agent reinforcement learning algorithm. It is an extension of the DDPG algorithm and is used to solve collaborative decision-making problems in multi-agent systems. Each agent has its own Actor and Critic network. The Actor network generates action strategies, and the Critic network evaluates the value of the strategy. The MADDPG algorithm enables agents to achieve collaborative decision-making and cooperation in complex multi-agent environments through collaborative learning and shared experience. The MADDPG algorithm has good performance and application in multi-agent collaborative tasks, multi-agent games and other fields.
[0084] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments. The above-described embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention.
Claims
1. A high-dimensional intelligent control model structure, characterized in that: include; The high-dimensional intelligent control model includes a hidden layer, an output layer, a cooperation state matrix ω configured to be hidden in the model structure, and a plurality of neurons arranged in a layer sequence; The ω structure plays a role in linking the global state neural network part and the multi-dimensional neural network; The hidden layers are set L 1 Layer, L 2 Layer and L 3 Layer, each hidden layer is set with neuron nodes; The information of each control dimension is composed of status information. And the composition structure of the action space Determine multiple control dimension information, according to the state information composition form in the control dimension information And the composition structure of the action space You can Expressed as Q value Right now 2. A high-dimensional intelligent control model structure training method, characterized in that: include; Acquire high-dimensional intelligent control model information, wherein each piece of high-dimensional intelligent control model information includes multiple control dimension information and feature information; Set the parameters of the evaluation network and the target network; Performing data analysis on the multiple control dimension information and feature information as well as the parameters of the evaluation network and the parameters of the target network to obtain a dual network output result, wherein the dual network output result includes an evaluation network output result and a target network output result; According to the dual network output results, the error calculation is performed to obtain the dual network error result, and the dual network error result is back-transferred to update the evaluation network parameters; The experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain training results.
3. The high-dimensional intelligent control model structure and training method according to claim 1, characterized in that: The loss function of the evaluation network includes: The loss function for evaluating the network is; Loss(θ)=E[||Drops x -Quevl x ||]. Qtar x ,Qevl x Represent the following two sets of vectors; According to the Bellman equation, we can get; in Represents the actions taken in K dimensions.
4. The high-dimensional intelligent control model structure and training method according to claim 1, characterized in that: The experience data (s, a, r, s′) is stored to construct an experience pool, and data is randomly sampled from the experience pool for training to obtain training results, including; The experience data (s,a,r,s′) is stored, where (s,a,r,s′)s is the initial environment state information, a is the action taken by the agent, r is the reward value obtained by the action, and s′ is the environment state information after action a acts on the environment. Small batch samples are randomly sampled for each step of training to obtain training results.