Nuclear power inspection method and device based on deep reinforcement learning, equipment and medium

Abstract

The application is suitable for the technical field of unmanned aerial vehicle cruising, and provides a nuclear power inspection method, device, equipment and medium based on deep reinforcement learning, which comprises the following steps: collecting state information containing the current position of the unmanned aerial vehicle at the current time, judging whether the distance between the current position and a target tracking point is less than a distance threshold, if yes, judging whether all target track points in a target inspection path have been tracked, if yes, completing the safety inspection of the nuclear power containment, otherwise, setting the target track point corresponding to the next time as the target tracking point, and jumping to the step of collecting the state information in real time, otherwise, according to the collected state information, obtaining the visual distance of the unmanned aerial vehicle at the current step by using a pre-trained optimal speed decision model, and tracking the target tracking point by using a preset trajectory tracking algorithm according to the visual distance and the current position, so that the average success rate and execution efficiency of the inspection task are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned aerial vehicle cruising, and particularly relates to a nuclear power inspection method, device, equipment and medium based on deep reinforcement learning. BACKGROUND

[0002] Energy is the key material basis for human survival and development, and is also the core focus of international politics, economy, military and diplomacy. The development of nuclear power brings a large amount of clean energy to the people of the country, but also brings potential threats of nuclear safety accidents. Periodic safety shell inspection of nuclear power plants is an important safety measure to ensure nuclear safety. However, the conventional surface detection method uses manual operation, which is high in cost and time-consuming. In recent years, with the development of unmanned aerial vehicle technology, the application of unmanned aerial vehicles (UAV) can improve detection efficiency, save time and cost, but due to the existence of Global Positioning System (GPS) signal missing area in the nuclear power plant scene, the inspection scheme based on GPS positioning signal will not be applicable to this scene. In the GPS missing scene, using Simultaneous Localization and Mapping (SLAM) algorithm for positioning is a common solution in the industry. However, the State-of-the-art (SOTA) SLAM algorithm, which is the best and most advanced algorithm, still has robustness problems in the performance inspection task of low-texture and single-structure building, and cannot guarantee the safety of unmanned aerial vehicle inspection. SUMMARY

[0003] The present application aims to provide a nuclear power inspection method, device, equipment and medium based on deep reinforcement learning, which aims to solve the problem of low execution efficiency and low success rate of unmanned aerial vehicle inspection due to the fact that the prior art cannot provide an effective nuclear power inspection method.

[0004] In one aspect, the present application provides a nuclear power inspection method based on deep reinforcement learning, which comprises the following steps:

[0005] Real-time acquisition of state information of the unmanned aerial vehicle, wherein the state information includes a current position of the unmanned aerial vehicle at the current time;

[0006] Judging whether the distance between the acquired current position and a target tracking point is less than a preset distance threshold, wherein the target tracking point is a certain target track point in a target inspection path planned in advance for a nuclear safety shell;

[0007] is, determining whether all target track points in the target inspection path have been tracked, if yes, completing the safety inspection of the nuclear containment, otherwise, setting a target track point corresponding to a next time corresponding to the current time in the target inspection path as a target tracking point, and jumping to the step of collecting the state information of the UAV in real time;

[0008] otherwise, according to the collected state information, using a pre-trained optimal speed decision model to obtain a visible distance of the UAV at the current step;

[0009] According to the visible distance and the current position, using a preset trajectory tracking algorithm to track the target tracking point, and jumping to the step of collecting the state information of the UAV in real time.

[0010] Preferably, before the step of using the pre-trained optimal speed decision model to obtain the visible distance of the UAV at the current step, the method further comprises:

[0011] In a first simulation environment built in advance, the optimal speed decision model is trained offline according to a pre-trained potential function set;

[0012] The first offline experience data set collected during the offline training of the optimal speed decision model is processed to obtain a real reward sample set;

[0013] According to the real reward sample set and a preset real reward predictor loss function, a real reward predictor is trained;

[0014] In a second simulation environment built in advance, the optimal speed decision model that has been trained offline is fine-tuned according to a real reward signal predicted by the trained real reward predictor.

[0015] Preferably, before the step of training the optimal speed decision model offline according to the pre-trained potential function set, the method further comprises:

[0016] In the first simulation environment, according to a pre-planned offline inspection path, a preset execution time of each step action, and a preset offline visible distance, a second offline experience data set of a simulation UAV in the task of executing the offline inspection path is collected;

[0017] According to the second offline experience data set, a preset first reward function, a preset potential function target formula, and a preset potential function update rule, the potential function is trained, and all the trained potential functions constitute the potential function set.

[0018] Preferably, the step of training the optimal speed decision model offline according to the pre-trained potential function set comprises:

[0019] screening the potential functions in the potential function set according to a preset screening rule to obtain a reward shaping potential function;

[0020] calculating a reward shaping term according to the reward shaping potential function and a preset reward shaping term formula;

[0021] training the optimal speed decision model offline according to the reward shaping term and a preset second reward function.

[0022] In another aspect, the present application provides a nuclear power plant inspection device based on deep reinforcement learning, which comprises:

[0023] an information collection unit configured to collect state information of a UAV in real time, wherein the state information comprises a current position of the UAV at a current time;

[0024] a first judgment unit configured to judge whether a distance between the current position collected and a target tracking point is less than a preset distance threshold, wherein the target tracking point is a target waypoint in a target inspection path planned in advance for a nuclear power plant containment;

[0025] a second judgment unit configured to judge whether all target waypoints in the target inspection path have been tracked if the distance between the current position collected and the target tracking point is less than the preset distance threshold, and if so, complete the safety inspection of the nuclear power plant containment, otherwise, set a target waypoint corresponding to a next time corresponding to the current time in the target inspection path as the target tracking point, and trigger the information collection unit to collect the state information of the UAV in real time;

[0026] a distance acquisition unit configured to acquire a visible distance of the UAV at a current step by using a pre-trained optimal speed decision model if the distance between the current position collected and the target tracking point is not less than the preset distance threshold; and

[0027] a target tracking unit configured to track the target tracking point by using a preset trajectory tracking algorithm according to the visible distance and the current position, and trigger the information collection unit to collect the state information of the UAV in real time.

[0028] Preferably, the device further comprises:

[0029] an offline training unit configured to train the optimal speed decision model offline according to a pre-trained potential function set in a first simulation environment built in advance;

[0030] a data processing unit, configured to process a first offline experience dataset collected in an offline training process of the optimal speed decision model to obtain a real reward sample set;

[0031] a predictor training unit, configured to train a real reward predictor according to the real reward sample set and a preset real reward predictor loss function; and

[0032] a fine-tuning training unit, configured to fine-tune the optimal speed decision model that has been trained offline according to a real reward signal predicted by the trained real reward predictor in a second simulation environment that has been built in advance.

[0033] Preferably, the apparatus further comprises:

[0034] an offline data collection unit, configured to collect a second offline experience dataset of a simulation unmanned aerial vehicle in a task of executing the offline inspection path according to a pre-planned offline inspection path, a preset execution time of each action and a preset offline visible distance in the first simulation environment; and

[0035] a potential function training unit, configured to train a potential function according to the second offline experience dataset, a preset first reward function, a preset potential function target formula and a preset potential function update rule, and all the trained potential functions constitute the potential function set.

[0036] Preferably, the offline training unit comprises:

[0037] a potential function screening unit, configured to screen the potential functions in the potential function set according to a preset screening rule to obtain a reward shaping potential function;

[0038] a reward shaping calculation unit, configured to calculate a reward shaping item according to the reward shaping potential function and a preset reward shaping item formula; and

[0039] an offline training sub-unit, configured to train the optimal speed decision model according to the reward shaping item and a preset second reward function.

[0040] In another aspect, the present application also provides a computing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned nuclear power inspection method based on deep reinforcement learning when executing the computer program.

[0041] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned nuclear power inspection method based on deep reinforcement learning.

[0042] The present application collects the state information including the current position of the unmanned aerial vehicle at the current time in real time, judges whether the distance between the current position and the target tracking point is less than the distance threshold, if yes, judges whether all target track points in the target inspection path have been tracked, if yes, completes the safety inspection of the nuclear power containment, otherwise, sets the target track point corresponding to the next time as the target tracking point, and jumps to the step of collecting the state information in real time until the tracking of all track points is completed, otherwise, according to the collected state information, uses the pre-trained optimal speed decision model to obtain the visible distance of the unmanned aerial vehicle at the current step, and uses the preset trajectory tracking algorithm to track the target tracking point according to the visible distance and the current position, thereby improving the average success rate and execution efficiency of the inspection task. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the implementation flowchart of the nuclear power inspection method based on deep reinforcement learning provided by the embodiment one of the present application;

[0044] Figure 2 is the flight guidance schematic diagram in the nuclear power inspection method based on deep reinforcement learning provided by the embodiment one of the present application;

[0045] Figure 3 is the implementation flowchart of the nuclear power inspection method based on deep reinforcement learning provided by the embodiment two of the present application;

[0046] Figure 4 is the system structure diagram of the first simulation environment in the nuclear power inspection method based on deep reinforcement learning provided by the embodiment two of the present application;

[0047] Figure 5 is the feature vector schematic diagram in the nuclear power inspection method based on deep reinforcement learning provided by the embodiment two of the present application;

[0048] Figure 6 is the implementation flowchart of the nuclear power inspection method based on deep reinforcement learning provided by the embodiment three of the present application;

[0049] Figure 7 is the implementation flowchart of the nuclear power inspection method based on deep reinforcement learning provided by the embodiment four of the present application;

[0050] Figure 8 is the structure schematic diagram of the nuclear power inspection device based on deep reinforcement learning provided by the embodiment five of the present application;

[0051] Figure 9 is a structural schematic diagram of a nuclear power plant inspection device based on deep reinforcement learning provided by an embodiment of the present application;

[0052] Figure 10 is a structural schematic diagram of a nuclear power plant inspection device based on deep reinforcement learning provided by an embodiment of the present application;

[0053] Figure 11 is a structural schematic diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0055] The specific implementation of the present application is described in detail below in combination with specific embodiments:

[0056] Example One:

[0057] Figure 1 An implementation process of a nuclear power plant inspection method based on deep reinforcement learning provided by an embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are as follows:

[0058] In step S101, the state information of the unmanned aerial vehicle is collected in real time, and the state information includes the current position of the unmanned aerial vehicle at the current time.

[0059] The embodiments of the present application are applicable to computing devices, such as personal computers, servers, etc. In the embodiments of the present application, according to the actual situation of the target building of the nuclear power plant safety shell to be inspected, a target inspection path is planned, and the target inspection trajectory is denoted as T: T=(g1,g2,…,g n ), wherein g t is the target track point of the unmanned aerial vehicle at time t, and p t ={x t ,y t ,z t} is the position of the unmanned aerial vehicle at time t. In the execution of the nuclear power plant safety shell inspection task by the unmanned aerial vehicle, the state information of the unmanned aerial vehicle is collected in real time, which includes first data obtained through the visual sensor of the unmanned aerial vehicle and second data obtained through the inertial sensor (IMU) of the unmanned aerial vehicle. The first data includes the image information of the photographed inspection target building, the current position p t of the unmanned aerial vehicle, and the attitude information of the unmanned aerial vehicle. The second data includes the bias information and the speed information of the IMU.

[0060] In step S102, it is determined whether the distance between the current position and the target tracking point is less than a preset distance threshold.

[0061] In this embodiment of the invention, the current position p of the drone is determined. t The target tracking point g corresponding to the current time t t If the distance between them is less than a preset distance threshold, then proceed to step S105; otherwise, proceed to step S103.

[0062] In step S103, based on the collected state information, the visible distance of the UAV in the current step is obtained using a pre-trained optimal speed decision model.

[0063] In this embodiment of the invention, the optimal speed decision model π(s) is a policy trained using a deep reinforcement learning (DRL) algorithm based on a proximal policy optimization (PPO) framework (i.e., employing a policy network-value network (Actor-Critic) architecture). Essentially, it is a neural network. The input and output of the optimal speed decision model are state and action, respectively. Therefore, the collected state information is processed into a state vector s. t , will s t The input is fed into the optimal velocity decision model π(s), which outputs the visible distance of the UAV in the current step.

[0064] In this embodiment of the invention, the training process of the optimal speed decision model is detailed in Embodiment 2 below, and will not be repeated here.

[0065] In step S104, the target tracking point is tracked using a preset trajectory tracking algorithm based on the visible distance and the current position.

[0066] In this embodiment of the invention, the UAV starts from the initial position p0 and moves towards the target track point g1 at the first moment to begin the inspection task. The target tracking point is the target inspection path T: T = (g1, g2, ..., g n For a given target waypoint in the map, at the first moment, the target tracking point is g1. During the inspection process, based on the visible distance... and current position p t The preset trajectory tracking algorithm is used to track the target point g. t Tracking is performed to complete the guidance and flight control for the current step. After each guidance and flight control step is completed, the process jumps to step S101 to continue execution, in order to determine whether the UAV has reached g at the current moment. t Specifically, such asFigure 2 As shown in the figure, a sphere with the current position p t of the UAV as the center is formed, and the target inspection path T is intersected with the sphere with a radius of r to obtain p t . t p t and p t determine the guidance direction of the UAV at time t, and the flight control part is responsible for completing the straight flight control of the UAV between p t and p t+1 in a fixed time (such as 1 second), so that the UAV reaches the position p t , wherein the trajectory tracking algorithm adopted is the line-of-sight (LOS) method.

[0067] In step S105, it is judged whether all target waypoints in the target inspection path have been tracked.

[0068] In the embodiment of the present application, when the distance between the current position p t of the UAV and the target tracking point g t+1 corresponding to the current time t is less than the preset distance threshold, it is indicated that the UAV has reached the target tracking point corresponding to the current time, and at this time, it is determined according to whether the target tracking point is the target waypoint corresponding to the last time in the target inspection path whether all target waypoints in the target inspection path have been tracked, and when all target waypoints have been tracked, step S107 is executed, otherwise, step S106 is executed.

[0069] In step S106, the target waypoint corresponding to the next time corresponding to the current time in the target inspection path is set as the target tracking point.

[0070] In the embodiment of the present application, the target waypoint g t+1 corresponding to the next time t+1 corresponding to the current time t in the target inspection path is set as the target tracking point for the next tracking, and the step S101 is jumped to continue to be executed until the tracking of all target waypoints is completed.

[0071] In step S107, the safety inspection of the nuclear power containment vessel is completed.

[0072] In the embodiment of the present application, when all target waypoints have been tracked, the safety inspection task of the nuclear power containment vessel is completed.

[0073] In the embodiment of the present application, the state information containing the current position of the unmanned aerial vehicle at the current time is collected in real time, it is judged whether the distance between the current position and the target tracking point is less than the distance threshold, yes, it is judged whether all target track points in the target inspection path have been tracked, yes, the safety inspection of the nuclear safety enclosure is completed, otherwise, the target track point corresponding to the next time is set as the target tracking point, and the step of collecting the state information in real time is jumped to until the tracking of all track points is completed, otherwise, according to the collected state information, the optimal speed decision model trained in advance is used to obtain the visible distance of the unmanned aerial vehicle at the current step, and according to the visible distance and the current position, the preset trajectory tracking algorithm is used to track the target tracking point, so that the average success rate and execution efficiency of the inspection task are improved.

[0074] Example Two:

[0075] Figure 3 The implementation process of the nuclear power inspection method based on deep reinforcement learning provided in the second embodiment of the present application is shown, only the part related to the embodiment of the present application is shown for the convenience of description, and the details are as follows:

[0076] In step S301, the optimal speed decision model is trained offline according to the pre-trained potential function set in the pre-built first simulation environment.

[0077] In the embodiment of the present application, the first simulation environment is composed of an open source four-rotor simulation environment Flightmare, a VI-SLAM system, a robot operating system (Robot Operating System, ROS) and a Unity rendering engine, and the data exchange between the components is realized through ROS communication.

[0078] Specifically, the building of the first simulation environment is realized through the following steps:

[0079] (1) the system structure of the first simulation environment is built;

[0080] In the embodiment of the present application, first, the Flightmare open source simulation environment is deployed, the conda environment running pytorch is deployed, and the open source SLAM algorithm (the present application adopts ORBSLAM3 algorithm) is run through, the above three modules are respectively run by 3 processes, then the communication code of data flow and unmanned aerial vehicle control information is written for the three modules, ROS communication is adopted to realize the communication of the three modules, Figure 4 The system structure of the first simulation environment is shown.

[0081] (2) the surface picture of the target building for inspecting the nuclear safety enclosure is obtained;

[0082] In this embodiment of the invention, in order to ensure the consistency of the shooting distance, a fixed shooting distance (e.g., 2m) is preset. Based on this shooting distance, a high-definition camera is used to photograph the surface of the target building of the nuclear power plant containment structure to obtain the original image.

[0083] (3) Preprocess the obtained original images;

[0084] In this embodiment of the invention, firstly, the true scale of the texture for rendering the 3D model of the target building is calculated based on the intrinsic parameters and shooting distance of the high-definition camera. Based on this true scale, the rendering scale of the 3D model corresponding to each original image is determined. Then, through manual screening, some special original images are filtered out, such as original images with obvious differences in brightness or original images with objects running across the screen. Finally, the K-means clustering algorithm is used to cluster the edges of the filtered original images. Specifically, edge features are extracted and clustered on the top, bottom, left, and right edges of each original image to provide constraints for random rendering.

[0085] (4) Based on the engineering drawings of the target building, the target building is modeled in the Unity engine, and the three-dimensional model of the target building is rendered using the pre-processed original image, rendering scale and edge constraint conditions to obtain the first simulation environment.

[0086] In this embodiment of the invention, the specific implementation method for offline training of the optimal velocity decision model based on a pre-trained potential function set is detailed in the following method embodiment, and will not be repeated here.

[0087] In step S302, the first offline experience dataset collected during the offline training of the optimal speed decision model is processed to obtain the real reward sample set.

[0088] In this embodiment of the invention, when processing the first offline experience dataset, the data processing is specifically implemented through the following steps:

[0089] (1) Data with relative pose error less than a preset error threshold (e.g., 0.4) in the first offline experience dataset collected during the offline training of the optimal velocity decision model will be filtered out to reduce noise in the first offline experience dataset and achieve the purpose of data cleaning.

[0090] (2) Extract the UAV pose estimation, UAV pose ground truth, velocity information, and IMU information from the first offline empirical dataset after data cleaning, and process these data into the following format: Figure 5 The eigenvector X′ shown t , X′ t It consists of three parts, namely the relative pose vector. Velocity vector v t And the IMU bias vector b t Wherein, P t est ∈R 3 Is the pose estimation value of the agent at time t, R 3 Indicates a three-dimensional vector, wherein each element belongs to the real number field R, Is the relative pose vector, Each element in Is the modulus of the relative pose of the pose estimation value P est With subscript i and j, The relative pose vector extracts the relative pose information of all pose estimation values within the window length l, and the window length of the present application is 5;

[0091] (3) All feature vectors constitute a real reward sample set.

[0092] In step S303, the real reward predictor is trained according to the real reward sample set and the preset real reward predictor loss function.

[0093] In the embodiment of the present application, the real reward predictor is used to solve the problem that the reward signal cannot be obtained due to the absence of GPS signal in the real environment, and the network structure adopts a full connection network. According to the real reward sample set and the real reward predictor loss function The real reward predictor is trained to train a reward predictor that does not depend on the positioning true value information, which can predict the reward signal in the real environment and achieve the effect of fine-tuning the optimal speed decision model in the real environment, wherein n is the sample number of the real reward sample set, y is the reward true value, Is the reward prediction value, Is the reward mean value, and w is the error weight. According to the weight formula The error weight is calculated, as described in the weight formula, which sets a higher weight for the real reward sample with a larger relative pose error. The weight of the real reward sample point with a target value less than or equal to the mean value is set to 1. The error weight can improve the fitting degree of the real reward predictor model for sample points with larger relative pose errors.

[0094] In step S304, in the second simulation environment built in advance, the optimal speed decision model trained offline is fine-tuned according to the real reward signal predicted by the trained real reward predictor.

[0095] In the embodiment of the present application, the second simulation environment is consistent with the building environment and system structure of the first simulation environment, and is used to simulate the real environment in which GPS is absent without obtaining the positioning true value information. In the real environment, the second flight path is planned offline, the optimal speed decision model pre-trained in the first simulation environment A is migrated to the second simulation environment B, in the second simulation environment B, the offline experience data set obtained by the simulation unmanned aerial vehicle cruising according to the planned second flight path is collected, and the optimal speed decision model is trained for a few rounds (for example, 10 rounds) of model training according to the offline experience data set and the reward signal predicted by the real reward predictor, so as to realize the fine tuning of the optimal speed decision model.

[0096] In the embodiment of the present application, in the first simulation environment built in advance, the optimal speed decision model is trained offline according to the pre-trained potential function set, the first offline experience data set collected is processed to obtain a real reward sample set, and the real reward predictor is trained according to the real reward sample set and the preset real reward predictor loss function, in the second simulation environment built in advance, the optimal speed decision model trained offline is fine tuned according to the real reward signal predicted by the trained real reward predictor, so that the optimal speed decision model can calculate the optimal visible distance according to the environment perceived by the unmanned aerial vehicle to determine the flight speed of the current step, effectively avoiding the positioning drift caused by too fast flight speed of the unmanned aerial vehicle in the area with sparse visual texture, and at the same time, in the area with rich visual texture, the model can make a decision to increase the flight speed to improve the efficiency of the inspection operation.

[0097] Example Three:

[0098] Figure 6 The implementation process of the nuclear power inspection method based on deep reinforcement learning provided in the embodiment three of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are as follows:

[0099] Before step S301 of the embodiment two is executed, the training of the potential function is realized by the following steps:

[0100] In step S601, in the first simulation environment, according to the pre-planned offline inspection path, the preset execution time of each step action and the preset offline visible distance, the second offline experience data set of the simulation unmanned aerial vehicle in the task of executing the offline inspection path is collected.

[0101] In the embodiment of the present application, in the first simulation environment built in advance, first, the offline inspection path is planned offline, and a plurality of uniform speed flight tasks with different speeds are set, and the offline inspection path corresponding to each flight task is respectively denoted as Then, the execution time T for each step of each flight mission. step And offline visibility distance R sight Configure (for example, configure T) step Set R to 1 second. sight (1 meter), in each flight mission T step and R sight If it is fixed, then the execution of each step of the drone's movement is in T. step Within a certain time, arrive at a distance R from the current position. sight The target point is determined by R. sight With T step Together, they determine the flight speed of the simulated drone for tracking the offline inspection path. In a constant-speed flight mission, the action a of each step... t They are all fixed, i.e., a. t =R sight They are all one-dimensional scalars.

[0102] When collecting the second offline experience dataset of simulated UAVs performing offline inspection path tasks, specifically, in the simulated UAVs performing offline inspection path tasks, at the current time t, the current position of the UAV is p. t The current state is s t The starting point of the current inspection path is g. t The endpoint is g t+1 According to a fixed R sight The LOS algorithm is used to guide the simulated UAV, such as... Figure 2 As shown, confirm the target point p of the current step. t+1 In T step Complete flight control within the time interval and transfer the state to s t+1 Arrival at s t+1 It can be based on the current state s t Calculate reward r t When the simulated drone arrives at p t+1 When, if p t+1 With g t+1 If the distance is less than the first threshold, the simulated drone is determined to have reached the destination, and the mission completion flag d is set. t =1, if p t+1 With g t and g t+1 If the straight-line distance between the determined flight paths exceeds the second threshold, it is considered a deviation from the flight path, and the current flight mission ends. Therefore, d t =1, otherwise d t =0, the drone will continue with the next action until it reaches g. t+1 When the drone performs action a t Arrival at s t+1Obtain a reward r t At time t, an experience exp t : exp t = (s t , a t , r t , d t ) is obtained, and the UAV arrives at g t from g t+1 A few hundred experiences will be generated in the process, and these experiences will constitute an episode, denoted as e t The UAV will generate hundreds of thousands of episodes in the process of performing a set of uniform flight tasks, and these episodes will constitute a set of offline data sets E = {e1, e2, …, e n}, and the offline data sets collected by all uniform flight tasks constitute the final second offline experience data set

[0103] In step S602, the potential function is trained according to the second offline experience data set, the preset first reward function, the preset potential function target formula, and the preset potential function update rule. All trained potential functions constitute a potential function set.

[0104] In the embodiments of the present application, the potential function adopts the update mode of the Critic network in the Actor-Critic framework. According to the second offline experience data set the preset first reward function the potential function target formula and the potential function update rule the potential function Φ θ (s t ) is trained, and each set of offline experience data corresponding to an offline potential function All trained potential functions constitute a potential function set, wherein r t and s t are the reward and state of the simulated UAV at time t, respectively, r e (s t , s t+1 ) is an error reward function, R g is the reward obtained by the simulated UAV when reaching the destination, R d is the penalty obtained by the simulated UAV when deviating from the route, δ is the target function of the potential function, θ is the network parameter, γ is the discount rate, λ is the preset factor, η is the learning rate, and l is the length of the state window.

[0105] In the embodiment of the present application, in the first simulation environment, according to the pre-planned offline inspection path, the preset execution time of each action and the preset offline visible distance, a second offline experience data set of the simulation unmanned aerial vehicle in the task of executing the offline inspection path is collected, the potential function is trained according to the second offline experience data set, the preset first reward function, the preset potential function target formula and the preset potential function updating rule, and all the trained potential functions constitute a potential function set, thereby solving the problem of sparse reward and enabling the optimal speed decision model in subsequent training to make a better decision.

[0106] Example Four:

[0107] Figure 7 The implementation process of the nuclear power inspection method based on deep reinforcement learning provided in the fourth embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are described as follows:

[0108] The offline training of the optimal speed decision model in step S301 of the second embodiment is implemented through the following steps:

[0109] In step S701, the potential functions in the potential function set are screened according to the preset screening rule, and a reward shaping potential function is obtained.

[0110] In the embodiment of the present application, the potential functions in the trained potential function set are screened according to the preset screening rule , and a reward shaping potential function Φ(s t ) is obtained.

[0111] When the potential functions in the potential function set are screened according to the preset screening rule, specifically, the screening of the potential functions is implemented through the following steps:

[0112] (1) According to the fixed action set corresponding to the potential function set θ , the probability p * of each action in the action set a θ output by the current behavior strategy π θ is calculated, wherein p θ is the action decision distribution output by π , the state space is represented by S, and the action space is represented by A.

[0113] (2) According to the calculated probability of each action, the preset first selection condition and the preset second screening condition, the potential function corresponding to the action with the highest probability is selected , that is, the potential function needs to meet the first screening condition and the second screening condition at the same time, wherein ​satisfy The first selection condition is representing π θ The probability of selecting action needs to be greater than a threshold value ∈1, which is used to measure the similarity between the current behavior policy and the offline uniform policy, and the second selection condition is representing π θ In the state window s t-l , s t-l+1 , …, s t of length l, the probability of selecting action needs to be greater than a threshold value ∈2, which is used to measure the similarity between the current behavior policy and the offline uniform policy in the short term. When the above two conditions are met, it is considered that the current behavior policy is similar to the offline uniform policy in action decision in the short term, and the inference error is within the allowable range.

[0114] (3) According to the screened potential function and the preset formula , the reward shaping potential function Φ(s t ) is obtained, that is, when the potential function that meets the condition is screened out, then When there is no potential function that meets the condition, the value of Φ(s t ) is 0.

[0115] In step S702, the reward shaping item is calculated according to the reward shaping potential function and the preset reward shaping item formula.

[0116] In the embodiment of the application, the reward shaping item f(s t , s t+1 ) is calculated according to the reward shaping potential function Φ(s t ) and the preset reward shaping item formula f(s t , s t+1 ) = γΦ(s t ) - Φ(s t+1 ).

[0117] In step S703, the optimal speed decision model is trained offline according to the reward shaping item and the preset second reward function.

[0118] In the embodiment of the application, the Actor and Critic networks of the optimal speed decision model adopt a fully connected network structure, the dimension of the input layer is determined by the state space S t , the hidden layer can be set to several layers (four layers are adopted in the application), and the dimension of the output layer is the dimension of the action space A.

[0119] The action space A combines the LOS algorithm, and path tracking using the LOS algorithm is divided into two parts of guidance and control. In the guidance part, a variable radius R sight is used to form a spherical surface with the current position p t as the center of the sphere, and the spherical surface intersects with the target trajectory T at p' sight . t The p t and p' t determine the guidance direction of the simulation UAV at time t, and the control part is responsible for the linear flight control of the simulation UAV between the two points p t and p' t . In the simulation UAV inspection process, the interval T step of each speed decision of the simulation UAV in the time sequence is fixed, so R sight can be used as a quantity related to the speed module, and the present application sets it as the action space: R represents a one-dimensional real number field.

[0120] Meanwhile, in the process of the simulation UAV performing the inspection task, the image of the simulation target building obtained by the binocular camera of the simulation UAV is used to obtain the visual feature information X t , the speed information v t and the bias information b t of the UAV obtained by the IMU of the simulation UAV, and the state space S t is composed of X t , v t and b t , that is, S t =(X t ,v t ,b t ), wherein X t : X W×H ∈{0,1} t is a two-dimensional visual feature distribution matrix of WxH, W is the width, H is the height, each element in the matrix corresponds to a pixel point in the image observed by the binocular camera at time t, and the value of the element is 0, indicating that the pixel is not a feature point, and 1, indicating that the pixel is a successfully matched feature point, v t : v 3 ∈R t is the flight speed at time t, and b t : b 3 ∈R

[0121] Specifically, the visual feature information X t is obtained by the following steps:

[0122] (1) obtaining an image of a simulation target building through a simulation unmanned aerial vehicle (UAV) binocular camera;

[0123] (2) performing visual feature matching on the image in a front end of an ORBSLAM3 system to obtain a first image;

[0124] (3) performing a pooling operation on the first image to obtain a second image;

[0125] (4) performing a flattening and zero-mean normalization operation on the second image to reduce the dimension of the visual feature, to obtain visual feature information X t , thereby solving the problem of sparsity of X t .

[0126] The preset second reward function is:

[0127] wherein,

[0128] R g is a reward obtained by the simulation UAV when reaching the end point, R d is a penalty obtained by the simulation UAV when deviating from the flight path, which is generally negative, and the judgment condition for reaching the end point can be set by setting an arrival threshold, when the distance between the position estimation value of the simulation UAV and the target waypoint is less than the arrival threshold, it is determined that the end point is reached, and the judgment condition for deviating from the flight path can be set by setting a deviation threshold, when the straight line distance between the position estimation of the simulation UAV and the current target flight path is greater than the deviation threshold, it is determined that the flight path is deviated;

[0129] w e is an error weight, r e (s t ,s t+1 ) is an error reward function, which is determined by the current state s t of the simulation UAV and the state s t after executing the action a t+1 , and is calculated according to the formula to obtain r e (s t ,s t+1 ), wherein, is a part of the state space s t of the simulation UAV at time t, p t ∈R 3 represents the pose information of the simulation UAV at time t, is the pose estimation of the simulation UAV at time t under the current state, which is the estimation value output by the SLAM system in real time, is the true value of the pose of the simulation UAV at time t, which is provided by the first simulation environment;

[0130] w g is a progress weight, rg (a t () represents progress rewards, according to the formula The calculation yielded that, To perform action a t The visible distance at that time;

[0131] f(s t ,s t+1 (This is for the body shaping item as a reward.)

[0132] Specifically, the offline training of the optimal speed decision model is achieved through the following steps:

[0133] (1) Based on the inspection tasks set in the first simulation environment, the agent interacts with the environment and collects experience e. t =(s t ,a t ,s t+1 ,d t ,prob t ), and add each collected experience to the experience cache pool E, where a t prob t From state s t The action sampled from the action distribution and the sampling probability of that action, d t To perform action a t State transition to s t+1 The task completion indicator at that time;

[0134] (2) When the number of experiences in the experience cache pool E reaches the preset threshold, calculate the value of each experience e in E. t reward signal r t Value V φ (s t ) and dominance value Among them, V φ (s t ) represents the output value of the Critic network with parameter φ. Through formula Calculated;

[0135] (3) Randomly sample an empirical batch of a batch size from E;

[0136] (4) Based on the empirical batch of this sampling, the calculated r t V φ (s t ), and the Critic network loss function Using gradient descent The Critic network is updated, where δ is the loss function of the Critic network, and the generalized advantage function is used as the value function V. φ’ The target value can reduce the variance of the gradient descent loss value and reduce the number of samples required for training;

[0137] (5) Similarly, based on the empirical batch of this sampling, the calculated r t V φ (s t ), and the loss function of the Actor network Using gradient descent The Actor network is updated, whereby... Let be the loss function of the Actor network, θ be the parameters of the Actor network, and E be the expectation of min(F1,F2). F1 is in s t Execute a in state t The score of the action, when (s t ,a t When the score is positive, the network parameters θ are adjusted to make the weights... The F2 algorithm increases the weight of the network parameter θ, and vice versa, decreases the weight of the network parameter θ, in order to achieve the goal of maximizing the expected total score. The F2 algorithm adds a clipping operation to the F1 algorithm to prevent the network parameter θ from changing too much, which would make the model training unstable.

[0138] (6) Repeat steps (1) to (5) above until the preset number of network updates is reached. Before each round of updates begins, save each experience in E and reset E.

[0139] In this embodiment of the invention, potential functions in the potential function set are filtered according to preset filtering rules to obtain reward shaping potential functions. Based on the reward shaping potential function and the preset reward shaping term formula, the reward shaping term is calculated. Based on the reward shaping term and the preset second reward function, the optimal speed decision model is trained offline, thereby enabling the optimal speed decision model to make optimal flight speed decisions in real time according to the scene conditions, improving the robustness of the positioning algorithm and the task execution efficiency of the UAV when performing inspection tasks in a simulation environment.

[0140] Example Five:

[0141] Figure 8 The structure of the nuclear power plant inspection device based on deep reinforcement learning provided in Embodiment 5 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, including:

[0142] The information collection unit 81 is configured to collect state information of the unmanned aerial vehicle in real time, and the state information includes a current position of the unmanned aerial vehicle at a current time;

[0143] The first judging unit 82 is configured to judge whether a distance between the collected current position and a target tracking point is less than a preset distance threshold, and the target tracking point is a target track point in a target inspection path planned in advance for the nuclear power containment;

[0144] The second judging unit 83 is configured to judge whether all target track points in the target inspection path have been tracked, and if yes, the safety inspection of the nuclear power containment is completed, otherwise, a target track point corresponding to a next time corresponding to the current time in the target inspection path is set as the target tracking point, and the information collection unit 81 is triggered to collect the state information of the unmanned aerial vehicle in real time;

[0145] The distance acquisition unit 84 is configured to acquire a visible distance of the unmanned aerial vehicle at the current step by using the pre-trained optimal speed decision model according to the collected state information; and

[0146] The target tracking unit 85 is configured to track the target tracking point by using a preset trajectory tracking algorithm according to the visible distance and the current position, and trigger the information collection unit 81 to collect the state information of the unmanned aerial vehicle in real time.

[0147] In the embodiment of the application, each unit of the nuclear power inspection device based on deep reinforcement learning can be realized by a corresponding hardware or software unit, each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the application. Specifically, the implementation of each unit can refer to the description of the first embodiment described above, which will not be repeated here.

[0148] Example Six:

[0149] Figure 9 The structure of the nuclear power inspection device based on deep reinforcement learning provided by the sixth embodiment of the application is shown, only the parts related to the embodiments of the application are shown for the convenience of description, which includes:

[0150] The offline training unit 91 is configured to perform offline training on the optimal speed decision model according to the pre-trained potential function set in the first simulation environment built in advance;

[0151] The data processing unit 92 is configured to perform data processing on the first offline experience data set collected in the process of performing offline training on the optimal speed decision model, to obtain a real reward sample set;

[0152] The predictor training unit 93 is configured to train the real reward predictor according to the real reward sample set and a preset real reward predictor loss function.

[0153] The fine-tuning training unit 94 is configured to fine-tune the optimal speed decision model trained offline according to the real reward signal predicted by the trained real reward predictor in the second simulation environment built in advance.

[0154] Preferably, the offline training unit 91 comprises:

[0155] The potential function screening unit 911 is configured to screen the potential functions in the potential function set according to a preset screening rule to obtain a reward shaping potential function.

[0156] The reward shaping calculation unit 912 is configured to calculate a reward shaping term according to the reward shaping potential function and a preset reward shaping term formula.

[0157] The offline training sub-unit 913 is configured to train the optimal speed decision model offline according to the reward shaping term and a preset second reward function.

[0158] In the embodiment of the present application, each unit of the nuclear power inspection device based on deep reinforcement learning can be realized by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application. Specifically, the implementation of each unit can refer to the description of the foregoing method embodiments, which will not be repeated here.

[0159] Example Seven:

[0160] Figure 10 The structure of the nuclear power inspection device based on deep reinforcement learning provided by the seventh embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of description, which comprises:

[0161] The offline data collection unit 101 is configured to collect a second offline experience data set of the simulation unmanned aerial vehicle in the task of executing the offline inspection path according to the pre-planned offline inspection path, the preset execution time of each action and the preset offline visible distance in the first simulation environment.

[0162] The potential function training unit 102 is configured to train the potential function according to the second offline experience data set, a preset first reward function, a preset potential function target formula and a preset potential function updating rule, and all the trained potential functions constitute a potential function set.

[0163] In the embodiments of the present application, each unit of the nuclear power inspection device based on deep reinforcement learning can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application. Specifically, the implementation of each unit can refer to the description of the foregoing method embodiments, which will not be repeated here.

[0164] Example Eight:

[0165] Figure 11 The structure of the computing device provided in the eighth embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for ease of illustration.

[0166] The computing device 11 of the embodiments of the present application includes a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. The processor 110 implements the steps of the foregoing method embodiments of the nuclear power inspection method based on deep reinforcement learning when executing the computer program 112, such as Figure 1 the steps S101 to S107 shown. Alternatively, the processor 110 implements the functions of each unit in the foregoing device embodiments when executing the computer program 112, such as Figure 8 the functions of the units 81 to 85 shown.

[0167] In the embodiments of the present application, the state information including the current position of the unmanned aerial vehicle at the current time is collected in real time, it is judged whether the distance between the current position and the target tracking point is less than the distance threshold, if yes, it is judged whether all target track points in the target inspection path have been tracked, if yes, the safety inspection of the nuclear power containment vessel is completed, otherwise, the target track point corresponding to the next time is set as the target tracking point, and the step of collecting the state information in real time is jumped to until the tracking of all track points is completed, otherwise, according to the collected state information, the optimal speed decision model pre-trained is used to obtain the visible distance of the unmanned aerial vehicle at the current step, and the target tracking point is tracked by using the preset trajectory tracking algorithm according to the visible distance and the current position, thereby improving the average success rate and execution efficiency of the inspection task.

[0168] The computing device of the embodiments of the present application can be a personal computer or a server. The steps implemented when the processor 110 in the computing device 11 implements a method of nuclear power inspection based on deep reinforcement learning when executing the computer program 112 can refer to the description of the foregoing method embodiments, which will not be repeated here.

[0169] Example Nine:

[0170] In the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned embodiment of the nuclear power plant inspection method based on deep reinforcement learning, for example, Figure 1 Or, the computer program is executed by the processor to implement the functions of the units in the above-mentioned embodiments of the device, for example Figure 8 Or, the computer program is executed by the processor to implement the functions of the units in the above-mentioned embodiments of the device, for example

[0171] In the embodiments of the present application, the state information containing the current position of the unmanned aerial vehicle at the current time is collected in real time, it is judged whether the distance between the current position and the target tracking point is less than the distance threshold, if yes, it is judged whether all the target track points in the target inspection path have been tracked, if yes, the safety inspection of the nuclear power containment is completed, otherwise, the target track point corresponding to the next time is set as the target tracking point, and the step of collecting the state information in real time is jumped to, until the tracking of all the track points is completed, otherwise, according to the collected state information, the optimal speed decision model pre-trained is used to obtain the visible distance of the unmanned aerial vehicle at the current step, according to the visible distance and the current position, the preset trajectory tracking algorithm is used to track the target tracking point, so as to improve the average success rate and the execution efficiency of the inspection task.

[0172] The computer readable storage medium of the embodiments of the present application can include any entity or device capable of carrying computer program code, recording medium, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0173] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A nuclear power plant inspection method based on deep reinforcement learning, characterized in that, The method comprises the following steps: Real-time acquisition of state information of the unmanned aerial vehicle, wherein the state information comprises a current position of the unmanned aerial vehicle at a current time; Determination of whether a distance between the current position and a target tracking point is less than a preset distance threshold, wherein the target tracking point is a target waypoint in a target inspection path planned in advance for a nuclear power containment; If yes, determination of whether all target waypoints in the target inspection path have been tracked, and if yes, completion of safety inspection of the nuclear power containment, and if not, setting of a target waypoint corresponding to a next time corresponding to the current time in the target inspection path as the target tracking point, and jumping to the step of real-time acquisition of state information of the unmanned aerial vehicle; Otherwise, according to the acquired state information, obtaining of a visible distance of the unmanned aerial vehicle at a current step by using a pre-trained optimal speed decision model; According to the visible distance and the current position, tracking of the target tracking point by using a preset trajectory tracking algorithm, and jumping to the step of real-time acquisition of state information of the unmanned aerial vehicle; Before the step of obtaining the visible distance of the unmanned aerial vehicle at the current step by using the pre-trained optimal speed decision model, the method further comprises: Offline training of the optimal speed decision model according to a pre-trained potential function set in a first simulation environment built in advance; Data processing of a first offline experience data set collected in the process of offline training of the optimal speed decision model to obtain a real reward sample set; Training of a real reward predictor according to the real reward sample set and a preset real reward predictor loss function; Fine-tuning training of the offline-trained optimal speed decision model according to a real reward signal predicted by the trained real reward predictor in a second simulation environment built in advance.

2. The method of claim 1, wherein, Before the step of offline training of the optimal speed decision model according to the pre-trained potential function set, the method further comprises: In the first simulation environment, collection of a second offline experience data set of a simulation unmanned aerial vehicle in the task of executing an offline inspection path according to the pre-planned offline inspection path, a preset execution time of each step action and a preset offline visible distance; Training of a potential function according to the second offline experience data set, a preset first reward function, a preset potential function target formula and a preset potential function update rule, wherein all the trained potential functions constitute the potential function set.

3. The method of claim 2, wherein, The step of offline training of the optimal speed decision model according to the pre-trained potential function set comprises: Screening of potential functions in the potential function set according to a preset screening rule to obtain a reward shaping potential function; Calculation of a reward shaping term according to the reward shaping potential function and a preset reward shaping term formula; Offline training of the optimal speed decision model according to the reward shaping term and a preset second reward function.

4. A nuclear power plant inspection device based on deep reinforcement learning, characterized in that, The device comprises: The information collection unit is configured to collect state information of the unmanned aerial vehicle in real time, wherein the state information comprises a current position of the unmanned aerial vehicle at a current time; The first judgment unit is configured to judge whether a distance between the current position and a target tracking point is less than a preset distance threshold, wherein the target tracking point is a target waypoint in a target tracking path planned for the nuclear power containment in advance; The second judgment unit is configured to judge whether all target waypoints in the target tracking path have been tracked, and if yes, complete the safety inspection of the nuclear power containment, and if not, set a target waypoint corresponding to a next time corresponding to the current time in the target tracking path as the target tracking point, and trigger the information collection unit to collect state information of the unmanned aerial vehicle in real time; The distance acquisition unit is configured to, if not, acquire a visible distance of the unmanned aerial vehicle at a current step by using a pre-trained optimal speed decision model according to the collected state information; and The target tracking unit is configured to track the target tracking point by using a preset trajectory tracking algorithm according to the visible distance and the current position, and trigger the information collection unit to collect state information of the unmanned aerial vehicle in real time. The device further comprises: The offline training unit is configured to perform offline training on the optimal speed decision model according to a pre-trained potential function set in a pre-built first simulation environment; The data processing unit is configured to perform data processing on a first offline experience data set collected in the process of performing offline training on the optimal speed decision model, to obtain a real reward sample set; The predictor training unit is configured to train a real reward predictor according to the real reward sample set and a preset real reward predictor loss function; and The fine-tuning training unit is configured to perform fine-tuning training on the optimal speed decision model that has been offline trained according to a real reward signal predicted by the trained real reward predictor in a pre-built second simulation environment.

5. The apparatus of claim 4, wherein, The device further comprises: The offline data collection unit is configured to collect a second offline experience data set of a simulation unmanned aerial vehicle in a task of performing an offline tracking path in the first simulation environment according to the pre-planned offline tracking path, a preset execution time of each step action and a preset offline visible distance; and The potential function training unit is configured to train potential functions according to the second offline experience data set, a preset first reward function, a preset potential function target formula and a preset potential function update rule, and all the trained potential functions constitute the potential function set.

6. The apparatus of claim 5, wherein, The offline training unit comprises: The potential function screening unit is configured to screen potential functions in the potential function set according to a preset screening rule, to obtain a reward shaping potential function; The reward shaping calculation unit is configured to calculate a reward shaping item according to the reward shaping potential function and a preset reward shaping item formula; and The offline training subunit is configured to perform offline training on the optimal speed decision model according to the reward shaping item and a preset second reward function.

7. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 3.

Images