A method for searching for a radioactive source based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's radioactive source search method, belong to nuclear safety technical field, steps are as follows: radiation field area is divided into several regions;Select a region and randomly generate radioactive source, and utilize agent in radiation field area to carry out autonomous search training, obtain initial training agent;From the remaining area that has not generated radioactive source, repeatedly select a region and randomly generate radioactive source, and again train update agent, obtain secondary training agent;The secondary training agent is placed in the radiation field area of randomly generated radioactive source and carries out autonomous search training, and according to reward value adjustment robot action after fusing the knowledge of each region, obtain global optimal autonomous search strategy;According to global optimal autonomous search strategy, search radioactive source in radiation field area.The application solves the problem that it is difficult to quickly and accurately locate radioactive source in complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of nuclear safety, and particularly relates to a radioactive source searching method based on reinforcement learning. BACKGROUND

[0002] With the wide application of nuclear technology in the fields of energy, medical treatment, industry and the like, the risk of nuclear leakage, loss of strong radioactive source and the like gradually increases, so that nuclear safety becomes a major issue of concern. The potential harmfulness and complexity of the nuclear environment determine that the related research is both crucial and extremely challenging. The fast and accurate searching of the lost or missing radioactive source is one of the key links of nuclear safety guarantee. Since the radioactive source can cause radiation to the human body, the searching by using artificial means is extremely harmful to the human body and low in efficiency. Therefore, with the maturity of the robot technology in recent years, the use of the robot to carry the radiation detector becomes a safer and more feasible searching scheme, and at present, many positioning algorithms are used to find the abnormal source existing due to the accident.

[0003] The prior art one designs a single radioactive source detection system using a drone to carry a detector, and provides a method for solving the three-dimensional positioning problem in the radioactive source scene. The prior art two improves the particle filtering algorithm by pre-setting the activity range of the radioactive source and introducing an optimized high-weight particle pair resampling algorithm, so as to improve the source positioning accuracy. The prior art three estimates the position and activity of the radioactive source by recording the radiation value change in the moving process of the robot and iteratively updating the particle distribution by using the particle filtering algorithm. The prior art four estimates the unknown source position in the environment by using different point radiation record values in space by using the recursive Bayesian estimation and the sequential Monte Carlo method. These methods need to know part of the source information in the initial stage, and the environment map is known or the radiation environment has no obstacle, but the actual radioactive source search task often faces the challenge of unknown environment map and complex terrain, and the obstacles existing in the environment will shield the radiation signal, which further increases the positioning difficulty. Therefore, the robot must have the ability of autonomously exploring the environment and intelligently planning the radiation measurement path. However, unlike the search method depending on the pre-defined path, the autonomous search strategy needs to fuse the current and historical radiation measurement information in real time, and dynamically decide the subsequent exploration path to efficiently locate the target.

[0004] In recent years, reinforcement learning algorithms have performed well in the field due to their advantages in optimizing decisions in uncertain environments. Researches from the field of autonomous search have shown good potential, but most studies have only been verified in relatively simple, obstacle-free radiation environments, and the adaptability to complex real-world scenarios is limited. Moreover, there are problems such as weak migration ability of the trained agent and poor environmental adaptability. Once the radiation scene changes, such as obstacle layout and source characteristics, it often takes a lot of time to retrain the model, and training a mature and effective agent usually requires a large number of samples and computing resources, with a high time cost. In actual radiation source search scenarios, time is extremely critical, and the risk of harm increases with the exposure of the source. Therefore, time efficiency, including training efficiency and online search efficiency, is the core bottleneck of the application landing in this field. Although some researches have focused on the migration ability or environmental adaptability of the agent, trying to make the trained agent locate and search for the source in the changed environment or unknown obstacle environment, there are still deficiencies in time efficiency and accuracy. SUMMARY

[0005] In view of the above problems in the prior art, the present application provides a radiation source search method based on reinforcement learning, which solves the problem of difficulty in quickly and accurately locating the radiation source in a complex environment.

[0006] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0007] The present application provides a radiation source search method based on reinforcement learning, comprising the following steps:

[0008] S1, dividing the radiation field area into several regions;

[0009] S2, selecting a region from the divided regions and randomly generating a radiation source in the region, and using the agent to perform autonomous source search training in the radiation field area to obtain an initially trained agent;

[0010] S3, selecting a region from the regions where no radiation source has been generated and randomly generating a radiation source in the region, and using the initially trained agent to perform autonomous source search training in the radiation field area, and after completing the autonomous source search training, obtaining an updated agent and entering S4;

[0011] S4, based on the agent updated last time, repeating S3 to update the agent again until each region has generated a radiation source and the corresponding autonomous source search training in each region is successful, and obtaining a twice trained agent;

[0012] S5, randomly generate a radiation source in the radiation field area, and place the secondary trained agent in the radiation field area for global autonomous search training, and adjust the robot action according to the reward value after fusing the knowledge of each area, to obtain a global optimal autonomous search strategy;

[0013] S6, according to the global optimal autonomous search strategy, the agent drives the robot to search for the radiation source in the radiation field area through the radiation detector.

[0014] Further, the radiation field area is divided into four areas, and the four areas are sorted in order from near to far and from horizontal to vertical according to the distance from the radiation field entrance, to obtain a first area, a second area, a third area and a fourth area.

[0015] Further, the S2 includes the following steps:

[0016] S21, selecting an area from the divided areas and randomly generating a radiation source in the area;

[0017] S22, based on the generated radiation source, using the agent to perform autonomous search training in the radiation field area until the first preset training number is reached or the autonomous search success rate of the agent reaches the first preset search success rate threshold, completing the autonomous search training, and obtaining an initial trained agent.

[0018] Further, the autonomous search training method includes the following steps:

[0019] A1, according to the current training round, selecting a radiation source generation range based on the progressive area strategy, and setting the training objective function of the agent according to the soft actor-critic network;

[0020] A2, based on the selected radiation source generation range, randomly generating a radiation source position, initializing the initial position of the robot and the soft actor-critic network, constructing the state space, the action space and the reward function;

[0021] A3, using the robot to obtain the radiation intensity measurement value at the current time through the radiation detector, and combining the robot position information and the historical sequence information as the state vector;

[0022] A4, inputting the state vector into the soft actor-critic network, and using the actor network to output continuous two-dimensional actions according to the training objective function of the agent after receiving the state vector, wherein the two-dimensional actions include the forward direction and the speed;

[0023] A5, updating the robot position by executing the two-dimensional action of the agent, and using the agent to evaluate the current strategy according to the reward function to reward or punish the agent or end the current training round;

[0024] A6, repeating A1-A5 until a complete training round ends, using the agent to store the experience tuples into the experience pool, wherein, represents the state at time step , represents the action at time step , represents the state at time step , represents the reward at time step , , , is a training end flag;

[0025] A7, repeating A1-A6, if the experience tuples in the experience pool reach a preset training capacity threshold, then entering A8;

[0026] A8, according to the data selection principle, using the agent to select experience samples from the experience pool for key sample training, and accumulating the training times and recording the success rate of the robot finding the radiation source as the autonomous source finding success rate, wherein the experience sample is the experience tuple corresponding to each training round of the agent.

[0027] Further, the speed in A4 adopts a radiation perception adaptive speed control mechanism;

[0028] The radiation perception adaptive speed control mechanism refers to that the agent dynamically adjusts the maximum speed of the robot according to the current radiation intensity measurement value.

[0029] Further, the data selection principle in A8 is to initialize the importance of the experience tuples according to the priority experience replay sampling probability of the experience tuples and sample from high to low.

[0030] Further, the method of iterative round training of key samples in A8 includes the following steps:

[0031] B1, based on the experience samples in the experience pool, sampling a batch of experience tuples and their corresponding importance weights from the experience pool, wherein, represents the current state in the sampled experience tuple, represents the current action in the sampled experience tuple, represents the current reward in the sampled experience tuple, represents the next step state in the sampled experience tuple;

[0032] B2, according to the next step state in the sampled experience tuple, sampling the next action from the current policy;

[0033] B3, calculating a target action value of the next action based on the sampled experience tuple and the selected next action;

[0034] The calculation expression of the target action value of the next action is as follows:

[0035] ,

[0036] Wherein, represents the target action value of the next action , represents a reward discount coefficient, represents taking the minimum value, represents the first action value output by the critic network about and , represents the second action value output by the critic network about and , represents a temperature coefficient, represents a logarithmic function, represents the probability of the actor network executing the next action ,

[0037] B4, updating the network parameters of the critic network based on the critic network loss function according to the target action value of the next action;

[0038] The calculation expression of the network parameters of the critic network is as follows:

[0039] ,

[0040] ,

[0041] Wherein, represents the network parameters of the critic network, represents an assignment, represents the learning rate of the critic network, represents the gradient of the critic network loss function, represents the critic network loss function, represents the mean square error;

[0042] B5, updating the network parameters of the actor network based on the actor network parameter loss function according to the sampled experience tuple and the selected next action;

[0043] The calculation expression of the network parameters of the actor network is as follows:

[0044] ,​

[0045] ,

[0046] wherein, denotes the network parameter of the actor network, denotes the learning rate of the actor network, denotes the gradient of the actor network parameter loss function, denotes the actor network parameter loss function;

[0047] B6, updating the temperature coefficient based on the temperature coefficient loss function according to the experience tuple obtained by sampling and the next action obtained by selection;

[0048] The calculation expression of the temperature coefficient is as follows:

[0049] ,

[0050]

[0051] wherein, denotes the learning rate of the temperature coefficient, denotes the gradient of the temperature coefficient loss function, denotes the temperature coefficient loss function, denotes the target entropy;

[0052] B7, calculating the time difference error and importance weight of the experience tuple based on the target action value of the next action, and updating the experience pool;

[0053] B8, updating the network parameter of the target critic network through the soft update model;

[0054] The calculation expression of the soft update model is as follows:

[0055] ,

[0056] wherein, denotes the network parameter of the target critic network, denotes the soft update coefficient;

[0057] B9, repeating B1-B8 for a preset key sample training round to complete the iterative round training of the key sample.

[0058] Further, the B7 comprises the following steps:

[0059] B71, calculating the time difference error of the experience tuple based on the target action value of the next action;

[0060] The calculation expression of the time difference error of the experience tuple is as follows:

[0061] ,

[0062] in, Indicates the first The temporal difference error of an empirical tuple Indicates time step The state below and actions The corresponding rewards Indicates time step The value of the actions output by the target commentator network. Indicates time step The value of action output from the actor network;

[0063] B72. Based on the temporal difference error of empirical tuples, dynamically allocate the priority of empirical samples for robot autonomous source finding;

[0064] The formula for calculating the priority of the empirical samples is as follows:

[0065] ,

[0066] in, Indicates the first Priority of each empirical sample This indicates taking the absolute value. Represents a differential positive constant;

[0067] B73. Based on the priority of the experience samples, calculate the priority experience replay sampling probability of the experience samples;

[0068] The formula for calculating the priority empirical replay sampling probability of the empirical sample is as follows:

[0069] ,

[0070] in, Indicates the first The probability of priority empirical replay sampling for each empirical sample. Indicates the first Prioritization of empirical samples after intensity adjustment Indicates the first Priority of empirical samples after intensity adjustment;

[0071] B74. Based on the priority experience replay sampling probability of experience samples, the importance weight of experience samples is calculated.

[0072] The expression for calculating the importance weight of the empirical samples is as follows:

[0073] ,

[0074] in, importance weight of a first experience sample of the robot autonomously searching for a source, representing the capacity of experience samples of the experience pool, representing the importance sampling compensation intensity hyperparameter, wherein,

[0075] S75, based on the importance weight of the experience sample, updating the importance weight of the experience tuple in the experience pool, completing the updating of the experience pool.

[0076] Further, the knowledge of each region in S5 is the optimal path experience learned by the agent when searching for the radioactive source in each region.

[0077] The beneficial effects of the present application are: the radioactive source searching method based on reinforcement learning provided by the present application is aimed at single robot positioning and searching for radioactive sources, provides a soft actor-critic network reinforcement SAC-CL learning framework integrating curriculum learning, and can realize autonomous source searching in a known boundary constrained radiation field; through simulation and physical experiment verification, the robot trained by the SAC-CL learning framework of the present scheme can efficiently position the radioactive source at any position within the limited boundary, and adapt to the dynamic environment with unknown obstacles, and has significant environmental adaptability and strategy migration ability; the present scheme only needs to know the prior information of the radiation field boundary, and can train the robot to complete the radioactive source position inference and optimal path planning through the SAC-CL learning framework, greatly reducing the exposure risk of personnel in the radiation high-risk environment; the present scheme optimizes the exploration efficiency and accuracy through the priority experience replay mechanism and the iterative round training of key samples, reduces the invalid wandering time of the robot in the radiation field, and prolongs the service life of the equipment. The current research only focuses on single source scene, and in the future, it will be expanded to more practical multi-radioactive source positioning and searching tasks.

[0078] Other advantages of the present application will be analyzed in more detail in the subsequent embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0080] Figure 1 The step flow chart of the radioactive source searching method based on reinforcement learning in the embodiments of the present application.

[0081] Figure 2 ​​This is a schematic diagram of the division of the course learning radiation field area in an embodiment of the present invention.

[0082] Figure 3 This is a schematic diagram of an intelligent agent driving a robot to search for a radiation source in an embodiment of the present invention.

[0083] Figure 4 This is a schematic diagram illustrating the information changes during the robot's search for a radioactive source in an embodiment of the present invention.

[0084] Figure 5 This is a schematic diagram of the course learning environment, priority experience playback, and soft actor-critic network structure in an embodiment of the present invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0086] Typically, lost or stolen radioactive sources are very small relative to the search area; therefore, this approach treats the radioactive source as a point source. In a radiation environment, there exists... decay, decay, Particles produced by decay and other processes such as neutron emission and nuclear fission, among which... decay Radiation has strong penetrating power, travels farther and has higher energy than other types of radiation, and is widely used in irradiation sterilization, industrial flaw detection, and radiotherapy. Therefore, most lost or stolen radiation sources are primarily of this type. The target of this search is a radioactive source. Radioactive source.

[0087] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a method for searching radioactive sources based on reinforcement learning, comprising the following steps:

[0088] In this scheme, in order to optimize the learning efficiency of the agent in the complex radiation field environment and avoid the difficulty of policy convergence caused by the too high initial task difficulty, a progressive curriculum learning (CL) mechanism is introduced. Inspired by the human progressive learning paradigm, the strategy decomposes the source seeking task into sub-stages with increasing difficulty through structured curriculum design, realizing the co-evolution of knowledge transfer and exploration ability.

[0089] S1, dividing the radiation field region into several regions;

[0090] As shown in Figure 2 , the radiation field region is divided into four regions, and the four regions are sorted in the order of near to far from the entrance of the radiation field and then horizontal to vertical, obtaining the first region , the second region , the third region and the fourth region . In this embodiment, the progressive curriculum learning mechanism takes the above four regions as an example. First, radioactive sources are generated from the four regions according to random turns, and then the global autonomous source seeking training of the agent driving the robot is performed, so as to realize the fusion of the divided region knowledge and obtain the globally optimal autonomous source seeking strategy.

[0091] S2, selecting a region from the divided regions and randomly generating a radioactive source in the region, and using the agent to perform autonomous source seeking training in the radiation field region to obtain an initially trained agent;

[0092] The S2 includes the following steps:

[0093] S21, selecting a region from the divided regions and randomly generating a radioactive source in the region;

[0094] S22, based on the generated radioactive source, using the agent to perform autonomous source seeking training in the radiation field region until the first preset training number is reached or the autonomous source seeking success rate of the agent reaches the first preset source seeking success rate threshold, completing the autonomous source seeking training and obtaining an initially pre-trained agent.

[0095] The method of autonomous source seeking training includes the following steps:

[0096] A1, according to the current training turn, selecting the radioactive source generation range based on the progressive region strategy, and setting the training objective function of the agent based on the soft actor-critic network;

[0097] The calculation expression of the training objective function of the agent is as follows:

[0098] ,

[0099] wherein, represents an optimal strategy of the agent searching for the radioactive source, represents a strategy of the agent searching for the radioactive source satisfying the maximum expected value, represents a state-action sequence trajectory sampled from the strategy of the agent searching for the radioactive source, represents an expectation of the state-action sequence trajectory, represents a reward discount coefficient at a time step , represents a time step at which the trajectory terminates, represents a state at a time step , represents an action at a time step , represents a reward at a time step , represents a temperature coefficient, represents a conditional entropy of the strategy at a state, represents a state belonging to , represents a state belonging to , , , represents a state belonging to

[0100] In the scheme, the reward discount coefficient is used to weigh the importance of future rewards, and the temperature coefficient is used to control the strength of random exploration, that is, the randomness of the output action; in the embodiment, the reward discount coefficient is set to 0.99;

[0101] In the scheme, the Soft Actor-Critic (SAC) algorithm additionally introduces the entropy of the strategy as a regularization term in addition to the standard cumulative reward maximization target, and adds the maximum entropy as a training target to make the agent obtain high cumulative rewards while avoiding falling into local optimum too early, increase the exploration of the algorithm, and alleviate the problem of insufficient search, and also enhance the generalization ability and robustness of the agent in different initial environments, so that the environmental adaptation ability of the agent is enhanced. The design of the training target function encourages the strategy to pursue high cumulative rewards while maintaining a certain randomness through high entropy. This mechanism has double advantages: on the one hand, it significantly enhances the exploration ability of the agent, prompting it to try new actions that may have higher long-term returns, effectively avoiding premature convergence of the strategy to a suboptimal solution; on the other hand, entropy regularization can automatically adjust the exploration intensity, naturally balancing the exploration of unknown information and the use of existing experience during the learning process.

[0102] Since a larger temperature coefficient will enhance the uncertainty of action distribution, the agent will explore more, while a smaller temperature coefficient will make the agent focus more on maximizing the immediate reward. A fixed temperature coefficient is difficult to adapt to the needs of different stages of the learning process, that is, the agent can explore more in the early stage, and in the later stage, as the strategy gradually optimizes and converges to the optimal solution, the exploration is reduced, and the strategy tends to be more deterministic. Therefore, an adaptive adjustment mechanism of the temperature parameter is introduced in the present scheme.

[0103] The calculation expression of the adaptive adjustment loss function of the temperature parameter is as follows:

[0104] ,

[0105] Wherein, represents the adaptive adjustment loss function of the temperature parameter, represents the parameterized strategy, represents the trajectory sampling expectation of the parameterized strategy , represents the logarithmic function, represents the probability of using action in state , represents the target entropy;

[0106] In the present scheme, based on the adaptive adjustment loss function of the temperature parameter, the dynamic adjustment of the temperature parameter can be realized. In the early stage of training, a larger temperature parameter can make the action distribution more uniform and strengthen the search. As the agent's understanding of the environment deepens and the strategy is gradually optimized, the temperature parameter decreases, the action randomness decreases, and the strategy finally converges to a high return area. When the temperature parameter tends to 0, the maximum entropy target degenerates into the standard reinforcement learning target.

[0107] The traditional uniform experience replay mechanism samples all experience samples with equal probability, which is difficult to efficiently use samples containing key information. For the learning efficiency of the reinforcement agent to the high-value state, the present scheme integrates a limited experience replay mechanism in the SAC framework, so as to dynamically allocate sample priorities by using the temporal difference error, so as to focus on experience with high learning potential through non-uniform sampling;

[0108] A2, based on the selected radioactive source generation range, randomly generating the position of the radioactive source, and initializing the initial position of the robot and the soft actor-critic network, constructing the state space, the action space and the reward function;

[0109] In the present embodiment, the calculation expression of the state space is as follows:

[0110] ,

[0111] ​wherein, represents a state space, represents a radiation value detected by the robot at the moment, represents a radiation change value detected by the robot between the current moment and the previous moment, represents a moving direction of the robot at the moment, represents a horizontal coordinate position of the robot at the moment, represents a vertical coordinate position of the robot at the moment, represents a historical action at the moment, represents a horizontal coordinate position of an obstacle, represents a vertical coordinate position of an obstacle, represents the current moment;

[0112] In the scheme, in order to make the reinforcement learning framework designed by the application suitable for the radiation field source finding task, the decision of the agent is mainly based on the environmental radiation information, therefore, the state space design contains the following key elements: the radiation value detected by the agent at the current moment and the current position, the moving direction information representing the motion state of the robot, the position information of the robot in the environment, and the position information of the obstacles within the perception range of the robot, the key information of the agent at the last three time steps is retained, including the radiation values at the detected positions at the last three time steps, and the actions performed. This design aims to help the agent to learn the implicit state more effectively, so as to make better subsequent decisions, while avoiding the computational burden and training instability caused by processing long sequences.

[0113] The calculation expression of the action space is as follows:

[0114] ,

[0115] wherein, represents a unit direction vector output by the actor network at the current moment, represents a unit speed of the robot in the horizontal coordinate direction, represents a unit speed of the robot in the vertical coordinate direction;

[0116] The calculation expression of the reward function is as follows:

[0117] ,

[0118] ,

[0119] ,

[0120] ,

[0121] wherein, represents a reward function, represents a sparse reward, represents an exploration reward, represents a radiation value exploration progress reward, represents a highest radiation value exploration reward, represents an obstacle penalty, represents an exploration time penalty, represents a flag bit indicating that the radioactive source is successfully found, represents a reward coefficient, represents a radiation intensity measurement value detected by the robot at the current moment, represents a radiation intensity measurement value detected by the robot at the last moment, represents a maximum radiation intensity measurement value from the history to the current moment, wherein, represents that the radioactive source is successfully detected, represents that the radioactive source is not detected;

[0122] In this embodiment, the radiation value exploration progress reward is positive when the radiation intensity measurement value detected by the robot is higher than the radiation intensity measurement value at the last moment, otherwise it is negative; the highest radiation value exploration reward is given when the radiation value detected by the robot is greater than the maximum radiation measurement value from the history to the current moment, otherwise it is 0; the obstacle penalty is given when the robot encounters an obstacle on the path from the current position to the position at the next moment, and the closer the distance, the greater the penalty, but since part of the optimal path may be close to the obstacle, therefore, in this embodiment, it only takes effect within a 1.2-meter area around the obstacle; the exploration time penalty is a balance item for the robot to find the radioactive source as soon as possible, and not to wander to cheat the reward.

[0123] A3, acquiring, by the robot, a radiation intensity measurement value at the current moment through a radiation detector, and combining robot position information and historical sequence information as a state vector;

[0124] A4, inputting the state vector into the soft actor-critic network, and outputting, by the actor network, a continuous two-dimensional action according to a training objective function of the agent after receiving the state vector, wherein the two-dimensional action includes a forward direction and a speed;

[0125] In this scheme, the action space is continuous and each dimension is independent, and the agent outputs a moving direction vector The continuous movement of the robot is controlled. In order to improve the source seeking efficiency of the robot, a radiation perception speed amplitude adaptive mechanism is designed: the speed amplitude at the next moment is dynamically determined according to the radiation value detected by the robot at the current moment according to a predetermined rule, and the speed is determined by the radiation intensity, that is, the moving distance of the robot is determined by the moving direction and the speed.

[0126] The speed in the A4 adopts a radiation perception adaptive speed control mechanism.

[0127] The radiation perception adaptive speed control mechanism means that the agent dynamically adjusts the maximum speed of the robot according to the current radiation intensity measurement value.

[0128] In this embodiment, the calculation expression of the maximum speed of the robot is as follows:

[0129] ,

[0130] Among them, The maximum speed of the robot is represented by Vmax, The radiation intensity measurement value at the current moment is represented by R.

[0131] In this embodiment, if the radiation intensity measurement value at the current moment is less than 1000CPM, that is, the robot is in a low radiation area, the robot adopts a maximum speed of 1 meter per second to quickly cross the area and accelerate the detection and search for the radioactive source, if the radiation intensity measurement value at the current moment is greater than or equal to 1000CPM and less than 5000CPM, that is, the robot is in a medium radiation area, the robot adopts a maximum speed of 0.8 meters per second to continue searching at a medium speed, and if the radiation intensity measurement value at the current moment is greater than or equal to 5000CPM, that is, the robot is in a high radiation area, the robot adopts a maximum speed of 0.6 meters per second to accurately position the radioactive source at a low speed.

[0132] The present scheme realizes the robot moving direction decision through the actor network learning, and realizes the maximum speed control through the radiation perception adaptive speed control mechanism, realizes the decoupling design between the moving direction and the speed, effectively reduces the learning complexity of the actor network, makes it focus on the direction decision, and helps to improve the stability of the training process.

[0133] As shown in Figure 3 , when the agent drives the robot to perform the radioactive source search task, the horizontal coordinate axis is the X axis, the vertical coordinate is the Y axis, the length unit of the horizontal and vertical coordinate axes is meter, and a radiation intensity degree diagram is attached, the robot enters the medium radiation area to search for the radioactive source after avoiding obstacles and quickly crossing the low radiation area, gradually moves to the high radiation area, enters the high radiation area to accurately position the radioactive source at a low speed, and finally reaches the position of the radioactive source.

[0134] As shown in Figure 4The agent drives the robot for the first 14 time steps, i.e., from time step to , in which the radiation intensity value detected by the detector is below 1000 , so the maximum speed for the first 14 steps is set to 1 meter per second; at the 15th step, i.e., at time step , the observed radiation intensity value is between 1000 CPM and 5000 CPM, so the maximum speed used by the action performed at the next time step is set to 0.8 meters per second. In the first few time steps from the starting point, such as , and , due to the accumulated environmental information, i.e., the radiation intensity value in the environment is small, the movement direction output by the actor network and the maximum speed set by the radiation-aware adaptive speed control mechanism jointly cause the actual movement distance of the robot to be relatively large, showing a more aggressive exploration behavior. As the accumulated observation point information increases, for example, after obtaining the radiation values at , and , it is found that the radiation intensity value at time is less than that at time and greater than that at time , so the uncertainty of the direction of the radiation source at time increases, and the actor network begins to adjust its output unit direction vector according to these historical information; at time , the movement distance of the robot is reduced to avoid moving further away from the source due to a wrong direction, which shows that the actor network becomes more cautious after learning from historical information; at time , the actor network integrates the current radiation intensity value and historical sequence information, such as the radiation intensity value at time , and the position of the robot, to infer that the current movement direction may cause the radiation intensity to decrease, i.e., to move away from the radiation source. Therefore, the actor network outputs a significantly different direction vector at time , which causes the movement trajectory of the robot at time to reverse direction, successfully correcting the path and approaching the radiation source. Through Figure 3 and Figure 4 , the effective execution of the radiation-aware adaptive speed control mechanism is verified, and the actor network combined with only three-step historical observation information can learn effective direction decision-making behavior, which together improves the autonomous source-finding performance.

[0135] A5, performing a two-dimensional action by the agent to update the robot position, while evaluating the current policy by the agent according to the reward function to reward or punish the agent or end the current training round;

[0136] The calculation expression of the robot position in the A5 is as follows:

[0137] ,

[0138] , ,

[0139] wherein, represents the robot position at the current time, represents the robot position at the last time, represents the maximum speed, represents the unit time step, represents the horizontal coordinate position of the robot at the current time, represents the vertical coordinate position of the robot at the current time, represents the horizontal coordinate position of the robot at the last time, represents the vertical coordinate position of the robot at the last time; in the embodiment, the unit time step is set to 1 second;

[0140] A6, repeating A1-A5 until a complete training round is ended, and then storing the experience tuple into the experience pool by the agent, wherein, represents the state at the time step , represents the action at the time step , represents the state at the time step , , represents the reward at the time step , is a training end flag; in the embodiment, , the training of the round is ended; the maximum training steps of a complete training round is 200, and the total training rounds is 8000.

[0141] A7, repeating A1-A6, if the experience tuples in the experience pool reach a preset training capacity threshold, then entering A8;

[0142] A8, according to the data selection principle, selecting experience samples from the experience pool by the agent to perform key sample training, and accumulating the training times and recording the success rate of the robot finding the radioactive source as the autonomous source finding success rate, wherein the experience sample is the experience tuple corresponding to each training round of the agent.

[0143] As Figure 5 shown, in the present scheme, the autonomous source-seeking training framework of the agent realizes progressive capability improvement through curriculum learning, state environment perception prompts the agent to make effective decisions, and the priority experience replay mechanism accelerates key sample learning, forming a complete "perception-decision-learning" closed-loop system.

[0144] The data selection principle in the A8 is to initialize the importance of the experience tuple according to the priority experience replay sampling probability of the experience tuple and sample from high to low.

[0145] In the present scheme, based on the data selection principle, the sample utilization rate can be improved, and the agent can learn high reward value decisions more effectively.

[0146] The method for iterative round training of key samples in the A8 includes the following steps:

[0147] B1, based on the experience samples in the experience pool, sampling a batch of experience tuples from the experience pool and their corresponding importance weights, wherein, s represents the current state in the sampled experience tuple, s represents the current action in the sampled experience tuple, s represents the current reward in the sampled experience tuple, s represents the next state in the sampled experience tuple;

[0148] B2, according to the next state in the sampled experience tuple, sampling a next action from the current policy;

[0149] B3, based on the sampled experience tuple and the selected next action, calculating the target action value of the next action;

[0150] The calculation expression of the target action value of the next action is as follows:

[0151] ,

[0152] wherein, s represents the target action value of the next action s represents the reward discount coefficient, s represents the minimum value, s represents the first action value output by the critic network about s and s represents the second action value output by the critic network about s and s represents the temperature coefficient, ​​​denotes a logarithmic function, denotes a strategy In the probability of performing the next action ;

[0153] B4, updating the network parameters of the critic network based on the critic network loss function according to the target action value of the next action;

[0154] The calculation expression of the network parameters of the critic network is as follows:

[0155] ,

[0156] ,

[0157] Wherein, denotes the network parameters of the critic network, denotes the assignment, denotes the learning rate of the critic network, denotes the gradient of the critic network loss function, denotes the critic network loss function, denotes the mean square error; in this embodiment, the learning rate of the critic network is set to ;

[0158] B5, updating the network parameters of the actor network based on the actor network parameter loss function according to the sampled experience tuple and the selected next action;

[0159] The calculation expression of the network parameters of the actor network is as follows:

[0160] ,

[0161] ,

[0162] Wherein, denotes the network parameters of the actor network, denotes the learning rate of the actor network, denotes the gradient of the actor network parameter loss function, denotes the actor network parameter loss function; in this embodiment, the learning rate of the actor network is set to ;

[0163] B6, updating the temperature coefficient based on the temperature coefficient loss function according to the sampled experience tuple and the selected next action;

[0164] The calculation expression of the temperature coefficient is as follows:

[0165] ,

[0166]

[0167] in, The learning rate represents the temperature coefficient. The gradient of the temperature coefficient loss function is represented. Represents the temperature coefficient loss function. Represents the target entropy;

[0168] B7. Based on the target action value of the next action, calculate the temporal difference error and importance weight of the experience tuple, and update the experience pool.

[0169] B7 includes the following steps:

[0170] B71. Calculate the temporal difference error of the empirical tuple based on the target action value of the next action;

[0171] The expression for calculating the temporal difference error of the empirical tuple is as follows:

[0172] ,

[0173] in, Indicates the first The temporal difference error of an empirical tuple Indicates time step The state below and actions The corresponding rewards Indicates time step The value of the actions output by the target commentator network. Indicates time step The action value output by the actor network; in this embodiment, it is achieved through a micro-positive constant. To prevent the sample from being accessed when the timing difference error is 0. Let be the action-value function of the critic network, corresponding to the critic network, and Let be the action-value function of the target critic network, corresponding to the target critic network;

[0174] B72. Based on the temporal difference error of empirical tuples, dynamically allocate the priority of empirical samples for robot autonomous source finding;

[0175] The formula for calculating the priority of the empirical samples is as follows:

[0176] ,

[0177] in, Indicates the first Priority of each empirical sample This indicates taking the absolute value. Represents a differential positive constant;

[0178] B73. Based on the priority of the experience samples, calculate the priority experience replay sampling probability of the experience samples;

[0179] The formula for calculating the priority empirical replay sampling probability of the empirical sample is as follows:

[0180] ,

[0181] in, Indicates the first The probability of priority empirical replay sampling for each empirical sample. Indicates the first Prioritization of empirical samples after intensity adjustment Indicates the first Priority of empirical samples after intensity adjustment;

[0182] Based on the priority of empirical sample replay sampling probability, it can be ensured that empirical samples with small temporal difference errors can still be sampled, as well as the diversity of samples during algorithm training; at the same time, in order to eliminate the distribution shift caused by priority sampling, importance weights are introduced in this scheme for bias correction.

[0183] B74. Based on the priority experience replay sampling probability of experience samples, the importance weight of experience samples is calculated.

[0184] The expression for calculating the importance weight of the empirical samples is as follows:

[0185] ,

[0186] in, The first step in indicating the robot's autonomous source finding The importance weight of each empirical sample This represents the size of the empirical sample pool. The hyperparameter representing the importance sampling compensation intensity is denoted by, where, In this embodiment, when At that time, no compensation was given for importance sampling. It fully compensates for the bias caused by sampling.

[0187] S75. Based on the importance weights of the experience samples, update the importance weights of the experience tuples in the experience pool to complete the update of the experience pool.

[0188] In the scheme, the priority experience replay mechanism is combined with the soft actor-critic algorithm to realize the cooperative optimization of the exploration efficiency of the action space and the state space, that is, the maximum entropy strategy drives sufficient random exploration of the action space to avoid premature convergence, and the priority experience replay mechanism guides the agent to focus on the key areas with higher information value in the state space for efficient experience mining, and through the combination of the two, the exploration efficiency of the autonomous source-seeking path of the robot and the convergence robustness of the final decision are significantly improved.

[0189] B8, updating the network parameters of the target critic network through a soft update model;

[0190] The calculation expression of the soft update model is as follows:

[0191] ,

[0192] wherein, represents the network parameters of the target critic network, represents a soft update coefficient;

[0193] B9, repeating B1-B8 for a preset number of key sample training rounds to complete the iterative round training of the key samples. In this embodiment, the preset number of key sample training rounds is 8000.

[0194] S3, selecting an area from the areas in which no radioactive source has been generated and randomly generating a radioactive source in the area, and using the initially trained agent to perform autonomous source-seeking training in the radiation field area, and after completing the autonomous source-seeking training, obtaining an updated agent and entering S4;

[0195] S4, based on the agent updated last time, repeating S3 to update the agent again until each area has generated a radioactive source and the corresponding autonomous source-seeking training in each area is successful, and obtaining a twice-trained agent;

[0196] In this embodiment, the remaining areas in the radiation field area except the areas in which radioactive sources have been generated are used as the secondary training areas. After generating a radioactive source in any one of the secondary training areas, the agent trained in the last round is used to perform autonomous source-seeking training in the radiation field area again until each divided area has generated a radioactive source and the corresponding autonomous source-seeking training is completed, and a twice-trained agent is obtained.

[0197] S5, randomly generating a radioactive source in the radiation field area and placing the twice-trained agent in the radiation field area for global autonomous source-seeking training, and adjusting the robot action according to the reward value after fusing the knowledge of each area to obtain a globally optimal autonomous source-seeking strategy;

[0198] The knowledge of each area in S5 refers to the optimal path experience learned by the agent when searching for the radioactive source in each area.

[0199] In the scheme, by fusing the optimal path learned when searching for the radioactive source in each area and combining the reward value to fine-tune the robot action, the agent can find a new path better than the original historical optimal path, thereby obtaining a global optimal source-seeking strategy for driving the robot to search for the radioactive source.

[0200] S6, according to the global optimal autonomous source-seeking strategy, the agent drives the robot to search for the radioactive source in the radiation detector through the radiation field area.

[0201] In the scheme, by making the agent execute the above radioactive source searching method, the early exploration range of the agent is limited, the exploration dimension is reduced, the learning of the basic navigation skill is accelerated, the search space is gradually expanded to force the policy update state distribution, and the local attractor domain is broken.

[0202] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for searching a radioactive source based on reinforcement learning, characterized in that, The method comprises the following steps: S1, dividing the radiation field area into several regions; S2, selecting a region from the divided regions and randomly generating a radioactive source in the region, and using an agent to perform autonomous source-seeking training in the radiation field area to obtain an initially trained agent; The autonomous source-seeking training method comprises the following steps: A1, according to the current training round, selecting a radioactive source generation range based on a gradual region strategy, and setting an agent training objective function based on a soft actor-critic network; A2, based on the selected radioactive source generation range, randomly generating a radioactive source position, initializing the initial position of the robot and the soft actor-critic network, and constructing a state space, an action space and a reward function; A3, using the robot to obtain the radiation intensity measurement value at the current time through the radiation detector, and combining the robot position information and the historical sequence information as a state vector; A4, inputting the state vector into the soft actor-critic network, and using the actor network to output a continuous two-dimensional action according to the agent training objective function after receiving the state vector, wherein the two-dimensional action includes a forward direction and a speed; A5, updating the robot position by executing the two-dimensional action by the agent, and using the agent to evaluate the current strategy according to the reward function to reward or punish the agent or end the current training round; A6、repeat A1~A5, until a complete training round ends, using the agent to store the experience tuple into the experience pool, wherein, represents the state at time step , represents the action at time step , represents the state at time step , , represents the reward at time step , is a training end flag; A7, repeating A1-A6, if the experience tuples in the experience pool reach a preset training capacity threshold, then entering A8; A8, using the agent to select experience samples from the experience pool according to the data selection principle for key sample training, and accumulating the training times and recording the success rate of the robot finding the radioactive source as the autonomous source-seeking success rate of the agent, wherein the experience sample is an experience tuple corresponding to each training round of the agent; S3, selecting a region from the regions where no radioactive source has been generated and randomly generating a radioactive source in the region, and using the initially trained agent to perform autonomous source-seeking training in the radiation field area, and after completing the autonomous source-seeking training, obtaining an updated agent and entering S4; S4, based on the agent updated last time, repeating S3 to update the agent again until each region has generated a radioactive source and the autonomous source-seeking training of each region is successful, and obtaining a twice trained agent; S5, randomly generating a radioactive source in the radiation field area, and placing the twice trained agent in the radiation field area for global autonomous source-seeking training, and adjusting the robot action according to the reward value after fusing the knowledge of each region to obtain a global optimal autonomous source-seeking strategy; S6, using the agent to drive the robot to search for the radioactive source in the radiation field area according to the global optimal autonomous source-seeking strategy.

2. The reinforcement learning-based radioactive source search method according to claim 1, wherein, The radiation field area is divided into four regions, and the four regions are sorted in order from near to far from the entrance of the radiation field and then horizontally to vertically, to obtain a first region, a second region, a third region and a fourth region.

3. The reinforcement learning-based radioactive source search method according to claim 2, wherein, The S2 comprises the following steps: S21, selecting a region from the divided regions and randomly generating a radioactive source in the region; S22, based on the generated radiation source, the intelligent agent is trained in the radiation field area until the first preset training times or the intelligent agent autonomous search success rate reaches the first preset search success rate threshold, the autonomous search training is completed, and the initial trained intelligent agent is obtained.

4. The reinforcement learning-based radioactive source search method according to claim 1, wherein, The speed in the A4 adopts a radiation-aware adaptive speed control mechanism. The radiation-aware adaptive speed control mechanism refers to that the intelligent agent dynamically adjusts the maximum speed of the robot according to the current radiation intensity measurement value.

5. The reinforcement learning-based radioactive source search method of claim 1, wherein, The data selection principle in the A8 is to initialize the importance of the experience tuple according to the priority experience replay sampling probability of the experience tuple and sample from high to low.

6. The reinforcement learning-based radioactive source search method according to claim 1, wherein, The method for iterative round training of key samples in the A8 includes the following steps: B1. sampling a batch of experience tuples from the experience pool based on experience samples in the experience pool and their corresponding importance weights, wherein, denotes a current state in the sampled experience tuple, denotes a current action in the sampled experience tuple, denotes a current reward in the sampled experience tuple, denotes a next state in the sampled experience tuple; B2, according to the next step state in the sampled experience tuple, a next action is sampled from the current policy; B3, based on the sampled experience tuple and the selected next action, the target action value of the next action is calculated; The calculation expression of the target action value of the next action is as follows: ， wherein represents the target action value of the next action , represents the reward discount factor represents taking the minimum value represents the critic network's estimate of the value of and the output of the first action value represents the critic network's estimate of the value of and the output of the second action value represents the temperature coefficient represents the logarithm function represents the policy performs the next action with the probability ; B4, according to the target action value of the next action, the network parameters of the critic network are updated based on the critic network loss function; The calculation expression of the network parameters of the critic network is as follows: ， ， wherein, represents a network parameter of the critic network, represents an assignment, represents a learning rate of the critic network, represents a gradient of the critic network loss function, represents a critic network loss function, represents a mean squared error; B5, according to the sampled experience tuple and the selected next action, the network parameters of the actor network are updated based on the actor network parameter loss function; The calculation expression of the network parameters of the actor network is as follows: ， ， wherein, network parameters of the actor network, a learning rate of the actor network, a gradient of the actor network parameter loss function, an actor network parameter loss function; B6, according to the sampled experience tuple and the selected next action, the temperature coefficient is updated based on the temperature coefficient loss function; The calculation expression of the temperature coefficient is as follows: ， wherein, denotes a learning rate of the temperature coefficient, denotes a gradient of the temperature coefficient loss function, denotes a temperature coefficient loss function, denotes a target entropy; B7, based on the target action value of the next action, the time difference error and importance weight of the experience tuple are calculated, and the experience pool is updated; B8, the network parameters of the target critic network are updated through the soft update model; The calculation expression of the soft update model is as follows: ， wherein, network parameters of the target critic network, denotes a soft update coefficient; B9, repeat B1-B8 for a preset number of key sample training rounds to complete the iterative round training of key samples.

7. The reinforcement learning-based radioactive source search method according to claim 6, wherein, The B7 includes the following steps: B71, based on the target action value of the next action, the time difference error of the experience tuple is calculated; The calculation expression of the time difference error of the experience tuple is as follows: ， wherein, denotes the time-difference error of the th experience tuple, denotes the state and action at time step the corresponding reward, denotes the action value of the target critic network output at time step denotes the action value output by the actor net at time step denotes the action value output by the actor net at time step . B72, based on the time difference error of the experience tuple, the priority of the experience sample of the robot autonomous search is dynamically allocated; The calculation expression of the priority of the experience sample is as follows: ， wherein, represents the priority of the i-th empirical sample, represents the priority of the i-th empirical sample, represents taking the absolute value, represents a small positive constant; B73, based on the priority of the experience sample, the priority experience replay sampling probability of the experience sample is calculated; The calculation expression of the priority experience replay sampling probability of the experience sample is as follows: ， in, Indicates the first The probability of priority empirical replay sampling for each empirical sample. Indicates the first Prioritization of empirical samples after intensity adjustment Indicates the first Priority of empirical samples after intensity adjustment; B74, based on the priority experience replay sampling probability of the experience sample, the importance weight of the experience sample is calculated; The calculation expression of the importance weight of the experience sample is as follows: ， wherein, represents the importance weight of the i-th experience sample in the autonomous source-seeking of the robot, represents the importance weight of the i-th experience sample in the autonomous source-seeking of the robot, represents the capacity of the experience pool, represents the importance sampling compensation strength hyperparameter, wherein, ; S75, based on the importance weight of the experience sample, the importance weight of the experience tuple in the experience pool is updated, and the update of the experience pool is completed.

8. The reinforcement learning-based radioactive source search method according to claim 7, characterized in that, The knowledge of each area in the S5 refers to the optimal path experience learned by the intelligent agent when searching for the radiation source in each area.

Images