Power transmission line robot motion control method and system based on DDPG algorithm

Abstract

The invention discloses a power transmission line robot motion control method and system based on a DDPG algorithm, and relates to the technical field of power transmission line inspection robots, and the method comprises the following steps: initializing a multi-task learning model of the DDPG algorithm, setting the initial weights of an inspection task and an obstacle clearing task, and starting multi-task learning; a knowledge migration mechanism is established, and features used for a multi-task learning model are extracted from existing line data and task experience to serve as initial input of a multi-task learning framework; and establishing an adaptive dynamic adjustment mechanism, adaptively adjusting discount factors of a DDPG algorithm according to real-time environment data including weather changes and line states, and optimizing an execution strategy of inspection and obstacle removal tasks. According to the method, a knowledge migration mechanism is established, useful features are extracted from existing line data and task experience and migrated to the current task, so that the robot can quickly adapt to a new power transmission line environment, the training time is shortened, and the task execution efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion control of a power transmission line inspection robot, and in particular to a power transmission line robot motion control method and system based on a DDPG algorithm. Background Art

[0002] With the continuous growth of electricity demand and the continuous expansion of the scale of power grids, the safety and reliability of transmission lines have become particularly important. During operation, transmission lines are easily affected by environmental factors (such as wind, rain, snow, lightning, etc.) and external interference (such as trees, buildings, birds, etc.), resulting in line failures and damage. Traditional transmission line inspection and maintenance methods mainly rely on manual labor, which is not only inefficient, but also has high safety risks. In recent years, with the rapid development of robotics and artificial intelligence technology, transmission line inspection robots have gradually become an efficient and safe alternative. Through automated inspection and maintenance by robots, the operating reliability and maintenance efficiency of transmission lines can be greatly improved.

[0003] However, there are still many challenges in motion control for existing power line inspection robots. For example, robots need to efficiently perform multiple tasks in complex and changing environments, such as inspection and obstacle removal. Traditional control algorithms are difficult to achieve optimal control in multi-task and dynamic environments. In addition, existing control algorithms have poor adaptability to environmental changes and it is difficult to dynamically adjust control strategies based on real-time environmental data, which affects the efficiency and accuracy of the robot's tasks. Therefore, there is an urgent need for a motion control method based on advanced algorithms to solve the above problems and improve the overall performance of power line inspection robots. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the problems existing in the above-mentioned existing transmission line robot motion control method based on DDPG algorithm and the system thereof, the present invention is proposed.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The transmission line robot motion control method based on the DDPG algorithm includes the following steps:

[0008] S1: Initialize the multi-task learning model of the DDPG algorithm, set the initial weights of the inspection task and obstacle removal task, and start multi-task learning;

[0009] S2: Establish a knowledge transfer mechanism to extract features for the multi-task learning model from existing line data and task experience as the initial input of the multi-task learning framework, so that the robot can quickly adapt to the new transmission line environment and reduce training time;

[0010] S3: Establish an adaptive dynamic adjustment mechanism to adaptively adjust the discount factor of the DDPG algorithm according to real-time environmental data, including weather changes and line status, optimize the execution strategy of inspection and obstacle removal tasks, and deploy the optimized multi-task learning model to the robot, so that the robot can perform efficiently in actual transmission line inspection.

[0011] As a preferred solution of the power transmission line robot motion control method based on the DDPG algorithm described in the present invention, wherein:

[0012] The multi-task learning model based on the DDPG algorithm adopts a shared parameter approach so that different tasks share a policy network and a value network, and are optimized through reward signals of different tasks during the training process;

[0013] Assume that the tasks of the power transmission line robot include inspection tasks and obstacle removal tasks. The expressions for defining the multi-task learning model include:

[0014] Generate a deterministic action in a given state, i.e., a shared policy network Actor: u(s|θ u ), where s represents the state, θ u Represents the parameters of the policy network;

[0015] Evaluate the value of a given state and action, i.e., the shared value network Critic: Q(s,a|θ Q ), where s represents the state, a represents the action, and θ Q Parameters representing the value network;

[0016] Multi-task loss function: For each task i, define a weight w i and the corresponding reward signal r i The total loss function is: L = ∑ i w i L i ;

[0017] Among them, L i is the loss function of task i, expressed as: L i =E[(r i +γQ(s',u(s|θ u )|θ Q )-Q(s,a|θ Q )) 2];

[0018] Among them, γ represents the discount factor, s' represents the next state, and r i is the reward signal for task i.

[0019] As a preferred solution of the power transmission line robot motion control method and system based on the DDPG algorithm of the present invention, the specific process of the multi-task learning is as follows:

[0020] Initialize the policy network parameters θ u and the value network parameter θ Q ;

[0021] Establish an independent experience replay buffer for each task to store the experience data of state, action, reward and next state;

[0022] In each training cycle, according to the task weight w i The ratio of,sample batches of data from the experience replay buffer of different tasks for training;

[0023] Use the gradient descent method to update the parameters of the shared policy network and the value network to minimize the multi-task loss function L;

[0024] Repeat the task sampling and parameter updating steps until convergence to complete the training of the multi-task learning model.

[0025] As a preferred solution of the power transmission line robot motion control method based on the DDPG algorithm of the present invention, the adaptive dynamic adjustment mechanism includes the following steps:

[0026] First, obtain the data change feedback value f(t);

[0027] Then dynamically adjust the discount factor:

[0028]

[0029] Among them, γ min and γ max are the minimum and maximum values ​​of the discount factor, β is the adjustment rate, and θ is the threshold of the feedback value;

[0030] Finally, the adjusted discount factor γ(t) is used to update the policy network and the value network. During the task execution, the robot selects action a according to the adjusted strategy.

[0031] As a preferred solution of the power transmission line robot motion control method and system based on the DDPG algorithm of the present invention, the calculation method of the data change feedback value f(t) is:

[0032] f(t)=α1ΔT(t)+α2ΔS(t);

[0033] Among them, T(t) is the weather change data and S(t) is the line status data, ΔT(t) and ΔS(t) are the weather change and line status change at time t respectively, and α1 and α2 are weight parameters.

[0034] As a preferred solution of the power transmission line robot motion control method and system based on the DDPG algorithm of the present invention, the method of updating the value network is: updating the value network parameter θ by minimizing the loss function Q ;

[0035] That is, the loss function expression is:

[0036] Where N is the batch size of randomly sampled experience data in the experience replay pool, y i is the target value, by which the network is updated;

[0037] y i =r t +γ(t)Q'(s t+1 ,u'(s t+1 )|θ Q' ), where Q' and u' are the target value network and target strategy network respectively.

[0038] As a preferred solution of the power transmission line robot motion control method and system based on the DDPG algorithm of the present invention, the update of the strategy network is based on the updated value network, that is, the parameter θ of the strategy network is updated by maximizing the expected reward output by the value network. u .

[0039] The power transmission line robot motion control system based on the DDPG algorithm is applied to the above-mentioned power transmission line robot motion control method based on the DDPG algorithm. The system includes the following functional modules:

[0040] Feature extraction and knowledge transfer module, which extracts useful features from historical data and task experience and transfers them to the current task, helping the robot quickly adapt to the new environment;

[0041] The multi-task learning module builds a multi-task learning model based on the DDPG algorithm. It starts multi-task learning by setting the initial weights of the inspection task and the obstacle removal task. During the model training process, it uses the loss functions of different tasks and adjusts the priority of the balance task through weights.

[0042] Adaptive dynamic adjustment module, which adaptively adjusts the discount factor of the DDPG algorithm according to real-time weather changes and line status changes, and optimizes the execution strategy of inspection and obstacle removal tasks;

[0043] The model training and evaluation module trains and evaluates the multi-task learning model to ensure the performance and reliability of the model.

[0044] Beneficial effects of the present invention:

[0045] 1. By introducing a multi-task learning mechanism, the present invention can process inspection and obstacle removal tasks at the same time, and achieve efficient coordination and optimization of multiple tasks by dynamically adjusting task weights. By designing an adaptive dynamic adjustment mechanism, the discount factor of the DDPG algorithm is dynamically adjusted according to real-time environmental data to ensure the adaptability and robustness of the robot in different environments, and optimize the execution strategies of inspection and obstacle removal tasks.

[0046] 2. The present invention establishes a knowledge transfer mechanism to extract useful features from existing line data and task experience and transfer them to the current task, so that the robot can quickly adapt to the new transmission line environment, reduce training time, and improve the efficiency of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0048] Figure 1 This is a flow chart of the power transmission line robot motion control method based on the DDPG algorithm proposed in the present invention;

[0049] Figure 2 It is a simulation diagram of the feedback value change in the execution of tasks in the power transmission line robot motion control method based on the DDPG algorithm proposed by the present invention;

[0050] Figure 3 This is a diagram showing the change in discount factor of the power transmission line robot motion control method based on the DDPG algorithm proposed in the present invention as the feedback value changes. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0054] Reference Figure 1 - Figure 2 , as an embodiment of the present invention, provides a transmission line robot motion control method and system based on DDPG algorithm, the method comprising the following steps:

[0055] S1: Initialize the multi-task learning model of the DDPG algorithm, set the initial weights of the inspection task and obstacle removal task, and start multi-task learning;

[0056] Specifically, the multi-task learning model based on the DDPG algorithm adopts a shared parameter approach so that different tasks share a policy network and a value network, and are optimized through reward signals of different tasks during the training process;

[0057] Assume that the tasks of the power transmission line robot include inspection tasks and obstacle removal tasks. The expressions for defining the multi-task learning model include:

[0058] Generate a deterministic action in a given state, i.e., a shared policy network Actor: u(s|θ u ), where s represents the state, θ u Represents the parameters of the policy network;

[0059] Evaluate the value of a given state and action, i.e., the shared value network Critic: Q(s,a|θ Q ), where s represents the state, a represents the action, and θ Q Parameters representing the value network;

[0060] Multi-task loss function: For each task i, define a weight w i and the corresponding reward signal r i The total loss function is: L = ∑ i w i L i ;

[0061] Among them, L iis the loss function of task i, expressed as: L i =E[(r i +γQ(s',u(s|θ u )|θ Q )-Q(s,a|θ Q )) 2 ];

[0062] Among them, γ represents the discount factor, s′ represents the next state, and r i is the reward signal for task i.

[0063] The specific process of multi-task learning is as follows:

[0064] Initialize the policy network parameters θ u and the value network parameter θ Q ;

[0065] Establish an independent experience replay buffer for each task to store the experience data of state, action, reward and next state;

[0066] In each training cycle, according to the task weight w i Assuming that the inspection task is of high importance, the initial w 巡检 Set to 0, 7, the initial weight w of the obstacle removal task 清除 Set to 0.3, and sample batches of data from the experience replay buffer of different tasks for training;

[0067] Use the gradient descent method to update the parameters of the shared policy network and the value network to minimize the multi-task loss function L;

[0068] Repeat the task sampling and parameter updating steps until convergence to complete the training of the multi-task learning model

[0069] S2: Establish a knowledge transfer mechanism to extract features for the multi-task learning model from existing line data and task experience as the initial input of the multi-task learning framework, so that the robot can quickly adapt to the new transmission line environment and reduce training time;

[0070] S3: Establish an adaptive dynamic adjustment mechanism to adaptively adjust the discount factor of the DDPG algorithm according to real-time environmental data, including weather changes and line status, optimize the execution strategy of inspection and obstacle removal tasks, and deploy the optimized multi-task learning model to the robot, so that the robot can perform efficiently in actual transmission line inspection.

[0071] Specifically, the adaptive dynamic adjustment mechanism includes the following steps:

[0072] First, obtain the data change feedback value f(t);

[0073] The calculation method of the data change feedback value f(t) is:

[0074] f(t)=α1ΔT(t)+α2ΔS(t);

[0075] Among them, T(t) is the weather change data and S(t) is the line status data, ΔT(t) and ΔS(t) are the weather change and line status change at time t respectively, and α1 and α2 are weight parameters.

[0076] Then dynamically adjust the discount factor:

[0077]

[0078] Among them, γ min and γ max are the minimum and maximum values ​​of the discount factor, β is the adjustment rate, and θ is the threshold of the feedback value;

[0079] Finally, the adjusted discount factor γ(t) is used to update the policy network and the value network. During the task execution, the robot selects action a according to the adjusted strategy.

[0080] In the inspection task, the adjustment of the discount factor affects the robot's trade-off between short-term and long-term rewards. When the discount factor is large, the robot pays more attention to long-term rewards, that is, ensuring the stability and health of the overall line status; when the discount factor is small, the robot pays more attention to short-term rewards, that is, solving current urgent problems. The adjustment of the discount factor affects the priority and execution strategy of the obstacle removal task. When the discount factor is large, the robot gives priority to removing obstacles that affect the safety of the line in the long term; when the discount factor is small, the robot gives priority to removing obstacles that currently affect the operation of the line.

[0081] In the above process, the method of updating the value network is: updating the value network parameter θ by minimizing the loss function Q ;

[0082] That is, the loss function expression is:

[0083] Where N is the batch size of randomly sampled experience data in the experience replay pool, y i is the target value, by which the network is updated;

[0084] y i =r t +γ(t)Q'(s t+1 ,u'(s t+1 )|θ Q' ), where Q' and u' are the target value network and target strategy network respectively.

[0085] The update of the policy network is based on the updated value network, that is, the parameters θ of the policy network are updated by maximizing the expected reward output by the value network. u

[0086] The power transmission line robot motion control system based on the DDPG algorithm is applied to the above-mentioned power transmission line robot motion control method based on the DDPG algorithm. The system includes the following functional modules:

[0087] The feature extraction and knowledge transfer module extracts useful features from historical data and task experience and transfers them to the current task to help the robot quickly adapt to the new environment. The multi-task learning module builds a multi-task learning model based on the DDPG algorithm. It starts multi-task learning by setting the initial weights of the inspection task and the obstacle removal task. During the model training process, the loss functions of different tasks are used to adjust the priority of the balance task through weight adjustment. The adaptive dynamic adjustment module adaptively adjusts the discount factor of the DDPG algorithm according to real-time weather changes and line status changes, and optimizes the execution strategy of the inspection and obstacle removal tasks. The model training and evaluation module trains and evaluates the multi-task learning model to ensure the performance and reliability of the model.

[0088] In summary, by introducing a multi-task learning mechanism, the present invention can simultaneously process inspection and obstacle removal tasks, and achieve efficient coordination and optimization of multiple tasks by dynamically adjusting task weights. By designing an adaptive dynamic adjustment mechanism, the discount factor of the DDPG algorithm is dynamically adjusted according to real-time environmental data to ensure the adaptability and robustness of the robot in different environments, optimize the execution strategies of inspection and obstacle removal tasks, and by establishing a knowledge transfer mechanism, extract useful features from existing line data and task experience and migrate them to the current task, so that the robot can quickly adapt to the new transmission line environment, reduce training time, and improve the efficiency of task execution.

[0089] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A transmission line robot motion control method based on DDPG algorithm, characterized in that: The following steps are involved: S1: Initialize the multi-task learning model of the DDPG algorithm, set the initial weights of the inspection task and obstacle removal task, and start multi-task learning; S2: Establish a knowledge transfer mechanism to extract features for the multi-task learning model from existing line data and task experience as the initial input of the multi-task learning framework, so that the robot can quickly adapt to the new transmission line environment and reduce training time; S3: Establish an adaptive dynamic adjustment mechanism to adaptively adjust the discount factor of the DDPG algorithm according to real-time environmental data, including weather changes and line status, optimize the execution strategy of inspection and obstacle removal tasks, and deploy the optimized multi-task learning model to the robot, so that the robot can perform efficiently in actual transmission line inspection.

2. The power transmission line robot motion control method based on the DDPG algorithm according to claim 1 is characterized in that: The multi-task learning model based on the DDPG algorithm adopts a shared parameter approach so that different tasks share a policy network and a value network, and are optimized through reward signals of different tasks during the training process; Assume that the tasks of the power transmission line robot include inspection tasks and obstacle removal tasks. The expressions for defining the multi-task learning model include: Generate a deterministic action in a given state, i.e., a shared policy network Actor: u(s|θ u ), where s represents the state, θ u Represents the parameters of the policy network; Evaluate the value of a given state and action, i.e., the shared value network Critic: Q(s,a|θ Q ), where s represents the state, a represents the action, and θ Q Parameters representing the value network; Multi-task loss function: For each task i, define a weight w i and the corresponding reward signal r i The total loss function is: L = ∑ i w i L i ; Among them, L i is the loss function of task i, expressed as: L i =E[(r i +γQ(s',u(s|θ u )|θ Q )- Q(s,a|θ Q )) 2 ]; Among them, γ represents the discount factor, s' represents the next state, and r i is the reward signal for task i.

3. The power transmission line robot motion control method based on the DDPG algorithm according to claim 2 is characterized in that: The specific process of the multi-task learning is as follows: Initialize the policy network parameters θ u and the value network parameter θ Q ; Establish an independent experience replay buffer for each task to store the experience data of state, action, reward and next state; In each training cycle, according to the task weight w i The ratio of,sample batches of data from the experience replay buffer of different tasks for training; Use the gradient descent method to update the parameters of the shared policy network and the value network to minimize the multi-task loss function L; Repeat the task sampling and parameter updating steps until convergence to complete the training of the multi-task learning model.

4. The power transmission line robot motion control method based on the DDPG algorithm according to claim 3 is characterized in that: The adaptive dynamic adjustment mechanism comprises the following steps: First, obtain the data change feedback value f(t); Then dynamically adjust the discount factor: Among them, γ min and γ max are the minimum and maximum values ​​of the discount factor, β is the adjustment rate, and θ is the threshold of the feedback value; Finally, the adjusted discount factor γ(t) is used to update the policy network and the value network. During the task execution, the robot selects action a according to the adjusted strategy.

5. The power transmission line robot motion control method based on the DDPG algorithm according to claim 4 is characterized in that: The calculation method of the data change feedback value f(t) is: f(t)=α1ΔT(t)+α2ΔS(t); Among them, T(t) is the weather change data and S(t) is the line status data, ΔT(t) and ΔS(t) are the weather change and line status change at time t respectively, and α1 and α2 are weight parameters.

6. The power transmission line robot motion control method based on the DDPG algorithm according to claim 5 is characterized in that: The method to update the value network is: update the value network parameters θ by minimizing the loss function Q ; That is, the loss function expression is: Where N is the batch size of randomly sampled experience data in the experience replay pool, y i is the target value, by which the network is updated; y i =r t +γ(t)Q'(s t+1 ,u'(s t+1 )|θ Q' ), where Q' and u' are the target value network and target strategy network respectively.

7. The power transmission line robot motion control method based on the DDPG algorithm according to claim 6 is characterized in that: The update of the policy network is based on the updated value network, that is, the parameters θ of the policy network are updated by maximizing the expected reward output by the value network u .

8. A power transmission line robot motion control system based on the DDPG algorithm, applied to the power transmission line robot motion control method based on the DDPG algorithm as claimed in claim 6, characterized in that: The system includes the following functional modules: Feature extraction and knowledge transfer module, which extracts useful features from historical data and task experience and transfers them to the current task, helping the robot quickly adapt to the new environment; The multi-task learning module builds a multi-task learning model based on the DDPG algorithm. It starts multi-task learning by setting the initial weights of the inspection task and the obstacle removal task. During the model training process, it uses the loss functions of different tasks and adjusts the priority of the balance task through weights. Adaptive dynamic adjustment module, which adaptively adjusts the discount factor of the DDPG algorithm according to real-time weather changes and line status changes, and optimizes the execution strategy of inspection and obstacle removal tasks; The model training and evaluation module trains and evaluates the multi-task learning model to ensure the performance and reliability of the model.

Images