A radiation perception path planning method based on deep Q network and curriculum learning

Abstract

The application discloses a radiation perception path planning method based on a deep Q network and course learning, and comprises the following steps: S1, initializing a radiation perception path planning environment and setting a reward function; S2, constructing a deep Q network agent and a priority experience replay mechanism; S3, training the deep Q network by using a course learning mechanism to obtain a trained deep Q network model; and S4, searching for a minimum radiation dose path on a map by using the trained deep Q network model and performing transfer learning to search for a minimum radiation dose path on a new map. By introducing the course learning mechanism, the priority experience replay, the reward function combining the radiation weight and the multi-factor punishment, and the transfer learning, an intelligent, self-adaptive and efficient path planning solution is provided for a nuclear radiation robot system, the safety, reliability and operation life of task execution are improved, and the method is suitable for checking, maintaining and emergency operation of high-risk nuclear facilities.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of path planning in nuclear radiation environment, and particularly relates to a radiation perception path planning method based on deep Q network and curriculum learning. BACKGROUND

[0002] Nuclear facilities need to perform frequent inspection, maintenance and emergency response operations in environments with varying radiation levels. The Fukushima Daiichi nuclear power plant disaster and other related incidents highlight the urgent need for robots systems that can operate in high-radiation environments, as these environments have strict limitations or are completely prohibited for human entry. In this case, autonomous or semi-autonomous robots can perform basic tasks including visual inspection, radiation mapping, sample collection and object manipulation, thereby minimizing human exposure to harmful radiation. However, traditional mobile robot path planning algorithms usually optimize path length, energy consumption or obstacle avoidance, without fully considering the unique challenges brought by the radiation field. The radiation spatial distribution in nuclear environments is often complex and uneven, with significant differences in radiation intensity in different regions. Therefore, the geometrically shortest path may expose the robot to unnecessary high cumulative radiation dose, potentially damaging electronic components and shortening the operating life. Therefore, it is necessary to plan a safe and efficient path for robots working in nuclear radiation environments for this type of robot system.

[0003] Path planning methods are generally divided into graph search-based path planning methods, sampling-based path planning methods and reinforcement learning-based path planning methods. Graph search algorithms represented by A* algorithm need to discretize map information into grids and perform domain search based on the grids. The degree of grid division directly affects the path planning accuracy of the algorithm, but higher accuracy will significantly increase the amount of calculation, thereby reducing the planning efficiency. The main advantage of A* algorithm is that it can guarantee to find the globally optimal path, but the path generated by it may contain more inflection points, affecting the path smoothness in actual application, and it is difficult to dynamically adapt to the nonlinear changes of the radiation distribution. Random sampling algorithms represented by RRT algorithm can be used for complex maps and can quickly generate locally optimal solutions. However, due to the randomness of the algorithm, the path generation is unstable and can only obtain a feasible solution, not guaranteeing to obtain an optimal solution, especially in high-dimensional radiation fields, the sampling efficiency is low, and the cumulative dose constraint cannot be effectively integrated. Reinforcement learning-based algorithms, such as standard deep Q network (DQN), learn strategies through agent interaction with the environment, and can handle uncertainty and dynamic environments, but traditional deep Q network has problems of unstable training, insufficient exploration and weak generalization ability in complex radiation fields, for example, the initial experience replay buffer can easily lead to suboptimal learning, and no special reward mechanism is designed for radiation exposure, causing the path optimization to be biased towards length and ignoring long-term radiation risk.

[0004] The aforementioned algorithms have achieved significant results in path obstacle avoidance, radiation dose minimization, and multi-robot collaboration, greatly improving the safety and intelligence of task execution in high-risk areas. However, these methods still have limitations such as low training efficiency, poor generalization, and insufficient radiation sensing. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a radiation-aware path planning method based on deep Q-networks and curriculum learning, which solves the problems of low training efficiency, poor generalization, and insufficient radiation perception in existing path planning methods.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a radiation sensing path planning method based on deep Q-networks and curriculum learning, comprising the following steps:

[0007] S1. Initialize the radiation sensing path planning environment and set the reward function;

[0008] S2. Construct a deep Q network proxy and a priority experience replay mechanism;

[0009] S3. The deep Q-network is trained using a course learning mechanism to obtain a trained deep Q-network model.

[0010] S4. The trained deep Q-network model is used to search for the lowest radiation dose path on the map, and transfer learning is performed to search for the lowest radiation dose path on the new map.

[0011] Furthermore: In S1, the reward function The specific expression is:

[0012]

[0013] In the formula, s This is the current state. For the new state The validity function, if If it is within the environmental boundary and there are no obstacles, then ,otherwise , This is an indicator function; it returns 1 if the condition is true, and 0 otherwise. For the target state, , For Manhattan distance, For cumulative dose, It is a positive hyperparameter. For radiation weight, High radiation threshold, for The radiation value, This is the "NOT" operator in logical operations.

[0014] The benefits of the above-mentioned further scheme are as follows: it designs a radiation-sensing reward system that integrates radiation dose threshold penalty and surrounding radiation status into the DQN framework, thereby minimizing cumulative exposure while ensuring path feasibility; and it supports transfer learning to quickly fine-tune the model and enhance generalization to new radiation environments.

[0015] Furthermore: In S2, the specific method for constructing a deep Q-network proxy is as follows:

[0016] The Q-network is initialized as a multilayer perceptron, with the input state and the output action value function estimate. The target network initially copies the Q-network parameters and updates the target network parameters with the Q-network parameters at fixed intervals.

[0017] Furthermore: In S2, the specific method for constructing the priority experience replay mechanism is as follows:

[0018] The initial priority of new experience is set to the current maximum value, and sampling is performed according to the priority probability distribution, with sampling probability... The specific expression is:

[0019]

[0020] In the formula, For experience i priority, For experience k priority, This is a priority control parameter. , For sampling experience i The timing difference error, It is a small constant.

[0021] Furthermore: In S3, the course learning mechanism includes an exploration phase and a course phase. The exploration phase generates a set of start-end pairs, which includes several start-end pairs. The course phase filters the set of start-end pairs according to a set Manhattan distance threshold, generating a filtered set of start-end pairs.

[0022] The beneficial effects of the above-mentioned further scheme are: to achieve progressive training through the course learning mechanism, thereby improving the model's adaptability from simple tasks to complex tasks.

[0023] Furthermore, in S3, the specific method for training the deep Q-network using the curriculum learning mechanism is as follows:

[0024] S31, based on the starting point-end point pair set generated in the current stage, a starting point-end point pair is randomly selected to set an environment, and an inner loop is performed based on the set environment, thereby completing the training of the deep Q network in the stage, wherein the method of performing one inner loop is specifically:

[0025] A1, updating the value of the greedy policy step by step , selecting an action according to the greedy policy, and performing the action to obtain a reward, a new state and a termination signal;

[0026] A2, storing the transition tuple containing the current state, action, reward, new state and termination signal into the experience replay buffer, and in response to the stored data in the experience replay buffer being greater than a preset number, selecting a batch size of experience values for one learning update;

[0027] A3, judging whether the end point is reached, if yes, updating the best reward, path and cumulative dose record, completing one inner loop, if not, increasing the time step by one and returning to A1;

[0028] S32, in response to completing the training of the deep Q network in all course stages, outputting the trained deep Q network model.

[0029] Further, in A1, the value of the greedy policy is controlled by a cosine decay scheduler, and the time step The expression of t is specifically:

[0030]

[0031] In the formula, is the minimum value of , is the maximum value of , is the number of time steps in the warm-up period, is the total number of time steps.

[0032] The above further scheme has the beneficial effects that: the priority experience replay and the cosine decay scheduler are introduced, the exploration and utilization balance is optimized, and the training stability and efficiency are improved.

[0033] Further, in A2, the method of selecting a batch size of experience values for one learning update is specifically:

[0034] B1, preferentially sampling a batch size of experience values from the experience replay buffer, and calculating the current Q value and the target Q value y

[0035]

[0036] In the formula, r is the reward,​​ a discount factor for rewards, for taking an action a new state resulting from, for taking a different action in the new state the maximum expected return that can be achieved, parameters of the target Q network, a termination signal;

[0037] B2, calculating a time-difference error from the current Q value y and the target Q value , and further constructing a loss function ;

[0038]

[0039]

[0040] wherein, parameters of the current Q network, computing expectations from a sampled batch of experiences from the prioritized replay buffer, importance sampling weights, B a batch size;

[0041] B3, optimizing the deep Q network by backpropagation.

[0042] Further, in S4, the method for searching the minimum radiation dose path on the map by the trained deep Q network model is specifically:

[0043] Based on the trained deep Q network model and the reward function, perform inference search on the grid map of the given start-end pair, adopt a greedy strategy to gradually decide actions in the search process, simulate the agent moving from the start point to the end point, and minimize the cumulative radiation exposure and path length to obtain the optimal path.

[0044] Further, in S4, the method for performing transfer learning is specifically:

[0045] Based on the trained deep Q network model parameters and the environment modeling framework, load the trained deep Q network model to the new agent, perform fine-tuning of the deep Q network model, inherit the source domain knowledge, and minimize the loss of the new domain to obtain the fine-tuned deep Q network model.

[0046] The present application has the following advantages:

[0047] (1) By introducing a curriculum learning mechanism, the training process is divided into an exploration phase and a gradual difficulty curriculum phase. This mechanism gradually transitions from simple short-path tasks to complex long-path tasks, significantly improving the agent's generalization ability and convergence speed. Compared with traditional DQN algorithms, the success rate of this method in the radiation environment is significantly improved, avoiding local optimal traps in early training and ensuring that the agent efficiently learns the optimal strategy under radiation constraints.

[0048] (2) The priority experience replay buffer dynamically adjusts the sampling priority based on the temporal difference error, focusing on high-value experience transfer, thereby optimizing learning efficiency and reducing the computational overhead of invalid samples. This design significantly reduces training time in complex grid maps, more effectively captures key dynamics of radiation exposure and path trade-offs, and improves the robustness of the algorithm in high-dimensional state space.

[0049] (3) The radiation weight reward function combined with multi-factor punishment seamlessly integrates radiation exposure into value estimation, guiding the agent to prefer low cumulative dose paths. Compared with classic path planning algorithms that ignore radiation (such as A* and RRT), the path generated by this invention has significantly reduced cumulative radiation dose, achieving a dynamic balance between path length and safety, and is suitable for high-risk scenarios such as nuclear environments.

[0050] (4) The cosine decay scheduler combined with a warm-up period ensures full map coverage in the early exploration phase and intensifies utilization in the later phase, enhancing the agent's adaptability to uneven radiation distribution. This mechanism, compared with the linear decay strategy, more smoothly transitions the exploration-exploitation trade-off, reduces the risk of overfitting, and improves the algorithm's generalization performance on dynamic tasks in multiple start-end pairs.

[0051] (5) The transfer learning framework supports parameter inheritance and selective fine-tuning of the source model, significantly reducing the training cost in new environments. Compared with traditional methods that train from scratch, this invention greatly shortens the adaptation time between different obstacle / radiation maps, enabling efficient cross-scene deployment and improving the robot's autonomous planning ability in complex radiation environments, reducing the need for human intervention. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of a radiation-aware path planning method based on deep Q network and curriculum learning.

[0053] Figure 2 An algorithm flowchart. DETAILED DESCRIPTION

[0054] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0055] As shown in Figure 1 and Figure 2 In one embodiment of the present application, a radiation-aware path planning method based on deep Q network and curriculum learning includes the following steps:

[0056] S1, initialize the radiation-aware path planning environment, set the reward function;

[0057] S2, construct a deep Q network agent and a priority experience replay mechanism;

[0058] S3, train the deep Q network using the curriculum learning mechanism to obtain a trained deep Q network model;

[0059] S4, search for the lowest radiation dose path on the map through the trained deep Q network model, and perform transfer learning to search for the lowest radiation dose path on the new map.

[0060] The idea of the present application is to provide a radiation-aware path planning method based on deep Q network and curriculum learning, aiming to provide an intelligent, adaptive and efficient path planning solution for nuclear radiation robot system, and ultimately improve the safety, reliability and operation life of task execution, suitable for inspection, maintenance and emergency operation of high-risk nuclear facilities.

[0061] In S1, the method for initializing the radiation-aware path planning environment is as follows:

[0062] Assume that the environment space is a two-dimensional grid containing obstacles and radiation dose distribution. By loading discrete obstacle array and radiation dose array (data obtained by Geant simulation of Co-60 radioactive source), the environment object is constructed. Define the state space as a fixed dimension vector, including the normalized coordinates of the current position and the target position, the obstacle information of the eight-direction neighborhood, the normalized radiation value of the current position and the radiation value of the eight-direction neighborhood.

[0063] In S1, the reward function considers Manhattan distance improvement, step penalty, radiation exposure penalty (linear scaling based on current dose and global weight) and high radiation threshold additional penalty, and the reward function The expression of the current state is specifically:

[0064]

[0065] In the formula, s is the current state, is the new state is the validity function, if is within the environmental boundary and has no obstacles, , otherwise When the environment is defined as a two-dimensional grid of , the state , x and y are the horizontal and vertical coordinates of the current state in the two-dimensional grid, and the obstacle set is O , then , and are the horizontal and vertical coordinates of the new state , is an indicator function, which is 1 if the condition is true, otherwise it is 0, is the target state, , is the Manhattan distance, is the cumulative dose, which is updated after each valid step, , is a positive hyperparameter, is the radiation weight, is the high radiation threshold, is the radiation value of , is the logical operation of "not", indicating that the new state is invalid, such as outside the environmental boundary or encountering obstacles, and a penalty is given.

[0066] where the Manhattan distance is improved: , which represents the change in distance from the current state to the target state, is a positive hyperparameter used to adjust the weight of this term. If is positive (i.e. closer to the target), the reward increases, indicating that the path has improved.

[0067] Step penalty: , which is used to penalize each step, possibly representing the consumption of some resources or the increase of time. This term is controlled by , where is a positive hyperparameter that determines the size of the penalty for each step, so as to avoid the agent taking too long a path.

[0068] Radiation exposure penalty: , It's a new state The radiation value represents the amount of radiation exposed under that condition. The weighting of radiation adjusts the degree of its influence on rewards. This is a positive hyperparameter, representing a discount factor used to balance current and future radiation exposure. This penalty is designed to incentivize agents to avoid high-radiation areas and reduce exposure.

[0069] Additional penalty for high radiation threshold: For exposure exceeding the high radiation threshold Additional penalties will be imposed on the state. It is an indicator function, when the new state... radiation value Exceeding the high radiation threshold When the value is 1, the indicator function is 1; otherwise, it is 0. It is a positive hyperparameter that determines the intensity of the additional penalty resulting from high radiation exposure.

[0070] In S2, the specific method for constructing a deep Q network proxy is as follows:

[0071] Initialize the Q-network as a multilayer perceptron, with state as input and action-value function estimation as output. The target network initially copies the Q network parameters and updates the target network parameters with the Q network parameters at fixed intervals.

[0072] In this embodiment, the deep Q-network structure includes five fully connected layers, supplemented by Xavier uniform initialization, layer normalization, ReLU activation function, and dropout regularization during training. The target network initially copies the parameters of the deep Q-network. and hard update at fixed intervals The optimizer uses the Adam algorithm with a learning rate of [missing information]. The discount factor takes the value of .

[0073] In S2, the specific method for constructing the priority experience replay mechanism is as follows:

[0074] The initial priority of new experience is set to the current maximum value, and sampling is performed according to the priority probability distribution, with sampling probability... The specific expression is:

[0075]

[0076] In the formula, For experience i priority, For experience k priority, This is a priority control parameter. , For sampling experience i The timing difference (TD) error, It is a small constant, usually taken as 0.01.

[0077] In this embodiment, the experience replay buffer adopts a priority format, with the initial priority of new experiences set to the current maximum value. Sampling is based on a priority probability distribution. Batch extraction is performed, and importance sampling weights are used. Correcting deviations, N This represents the total number of experiences in the playback buffer. This priority mechanism mathematically reduces the estimation variance: the sampling bias of standard playback leads to... , This is an estimate of the Q value, while priority sampling is achieved by focusing on high... Experience makes This accelerates convergence.

[0078] In S3, the learning mechanism includes an exploration phase and a learning phase. The exploration phase generates a set of start-endpoint pairs, containing several pairs with no distance restrictions. The learning phase filters these start-endpoint pairs based on a set Manhattan distance threshold, generating a filtered set that ensures the start and end points are located in non-obstacle positions. This mechanism dynamically adjusts task complexity, facilitating the agent's gradual transition from learning simple short paths to learning complex long paths.

[0079] In S3, the specific method for training the deep Q-network using the curriculum learning mechanism is as follows:

[0080] S31. Based on the start-endpoint pair set generated in the current stage, randomly select start-endpoint pairs to set the environment, and perform an inner loop based on the set environment to complete the training of the deep Q-network in this stage. The specific method for performing one inner loop is as follows:

[0081] A1. Update the greedy strategy step-by-step (episode). The value is determined by a greedy strategy, which selects actions and executes those actions to obtain rewards, new states, and termination signals.

[0082] A2. Store the transition tuple containing the current state, action, reward, new state, and termination signal into the experience replay buffer. If the data stored in the experience replay buffer exceeds the preset number, select the experience value of the batch size for a learning update.

[0083] A3. Determine if the destination has been reached. If yes, update the best reward, path, and cumulative dose records, and complete one inner loop. If no, increment the time step by one and return to A1.

[0084] S32, in response to completing the training of all course stages of the deep Q network, outputting the trained deep Q network model. After the training is completed, the overall success rate and exploration coverage are calculated, and the model and the optimal path are saved.

[0085] In A1, the greedy strategy of the deep Q network is controlled by a cosine decay scheduler, and the time step t is The expression of is specifically:

[0086]

[0087] In the formula, is the minimum value of , is the maximum value of , is the number of time steps in the warm-up period, is the total number of time steps.

[0088] In the greedy strategy of the deep Q network, controls the balance between exploration and utilization: a higher encourages random actions to explore the state-action space, while a lower prefers the optimal action based on Q value to utilize the learned knowledge. In order to achieve a smooth transition and avoid the sudden change problem caused by linear decay (such as insufficient exploration in the early stage or too fast convergence in the later stage), the embodiment adopts a cosine decay scheduler with a warm-up period for control. The scheduler is only applied after the end of the warm-up period; during the warm-up period, , usually linearly decreases from the initial value to . This design is based on the principle of cosine annealing, which ensures that is smoothly decayed from to , and finally reaches the minimum value at the total training step. Using this cosine decay scheduler can reduce gradient oscillation and improve convergence stability, thereby ensuring that the exploration-exploitation balance is more effective in complex environments such as radiation perception path planning.

[0089] In A2, the method of selecting a batch size of experience values for one learning update is specifically:

[0090] B1, sampling a batch size of experience values from the experience replay buffer, calculating the current Q value and the target Q value y ;

[0091]

[0092] In the formula, r is the reward, reward discount factor, in the embodiment , action to take new state resulting from taking the action maximum expected reward obtainable by taking a different action in the new state parameters of the target Q network termination signal

[0093] B2, a temporal difference error is calculated from the current Q value and the target Q value y , and a loss function is constructed ;

[0094]

[0095]

[0096] wherein parameters of the current Q network is computed from the sampled batch of experiences in the prioritized replay buffer is the importance sampling weight B is the batch size

[0097] B3, the deep Q network is optimized by backpropagation.

[0098] In S4, the method for searching for the minimum radiation dose path on the map by the trained deep Q network model is specifically:

[0099] Based on the trained deep Q network model and the reward function, an inference search is performed on a grid map of a given start-end pair, and a greedy strategy is adopted to gradually decide actions in the search process, simulating the movement of an agent from the start point to the end point, and minimizing the cumulative radiation exposure and path length to obtain an optimal path.

[0100] In this embodiment, the search is based on the generalization ability improved by curriculum learning, ensuring the tendency to low-cost areas in complex radiation fields, and ultimately realizing efficient path planning under radiation constraints.

[0101] In S4, the method for performing transfer learning is specifically:

[0102] ​​Based on the trained deep Q network model parameters and the environment modeling framework, the trained deep Q network model is loaded to a new agent, fine-tuning of the deep Q network model is performed to inherit the source domain knowledge and minimize the loss of the new domain, and a fine-tuned deep Q network model is obtained. Specifically, according to a fine-tuning strategy, part of the parameters is frozen and the learning rate is reduced, and the method of S3 is repeated to train the deep Q network, so as to refine the decision strategy of the agent in the new environment. Subsequently, a greedy reasoning process using the fine-tuned model is performed to search for a path from the starting point to the ending point on the new map. This step is based on the domain adaptation theory and outputs a low-radiation exposure path sequence on the new map, and finally realizes cross-domain radiation constraint path planning.

[0103] In summary, the present application significantly optimizes the DQN framework by introducing mechanisms such as course learning phased progressive training (gradual transition from simple short distance tasks to complex long distance tasks), priority experience replay (focusing on high TD error experience to improve sampling efficiency), and transfer learning (fine-tuning from source radiation map to target map, supporting hierarchical freezing to quickly adapt to new environment). Compared with the A* algorithm, the present method does not rely on fine grid division, avoids computational explosion, and generates smooth paths through learning, while dynamically incorporating radiation weight penalties to reduce cumulative dose by more than 20%. Compared with the RRT algorithm, the present method reduces randomness dependence, provides more stable near-global optimal solution, and samples more efficiently in complex radiation distribution. Compared with the standard DQN, the present application improves training stability and generalization ability through special radiation-aware reward design (including peripheral radiation state representation and high threshold penalty) and exploration optimization (such as epsilon cosine decay scheduler), and successfully increases the success rate to more than 90%, which is suitable for real-time autonomous navigation of nuclear facilities.

[0104] In the description of the present application, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implying the number of technical features indicated. Therefore, the features limited by "first", "second", "third" can explicitly or implicitly include one or more of the features.

Claims

1. A radiation-aware path planning method based on deep Q-network and curriculum learning, characterized in that, The method comprises the following steps: S1, initializing a radiation-aware path planning environment, and setting a reward function; S2, constructing a deep Q network agent and a priority experience replay mechanism; S3, training the deep Q network by using a curriculum learning mechanism to obtain a trained deep Q network model; S4, searching for a minimum radiation dose path on a map by using the trained deep Q network model, and performing transfer learning to search for a minimum radiation dose path on a new map; In S1, the reward function is expressed as: where s is the current state, is the new state is the validity function, if is inside the environment boundaries and free of obstacles, then , otherwise , is the indicator function, 1 if the condition is true, 0 otherwise, is the target state, , is the Manhattan distance, is the cumulative dose, is the positive hyperparameter, is the radiation weight, is the high radiation threshold, is the radiation value of , is the logical negation in the operation. In S3, the curriculum learning mechanism comprises an exploration stage and a curriculum stage, the exploration stage generates a set of start point-end point pairs, and the curriculum stage filters the set of start point-end point pairs according to a set Manhattan distance threshold to generate a filtered set of start point-end point pairs.

2. The method of claim 1, wherein, In S2, the method for constructing the deep Q network agent is specifically as follows: The Q network is initialized as a multi-layer perception machine, an input state is input, an action value function estimate is output, the target network is initially copied with the Q network parameters, and the target network parameters are updated at a fixed interval according to the Q network parameter update.

3. The method of claim 1, wherein, In S2, the method for constructing the priority experience replay mechanism is specifically as follows: The new experience initial priority is set to the current maximum value, and is sampled according to a priority probability distribution, and the sampling probability The expression is specifically: wherein is an empirical i priority, is an empirical k priority, is a priority control parameter, , is a timing differential error i empirically sampled, is a small constant.

4. The method of claim 1, wherein, In S3, the method for training the deep Q network by using the curriculum learning mechanism is specifically as follows: S31, based on the set of start point-end point pairs generated in the current stage, a start point-end point pair is randomly selected to set an environment, and an inner loop is performed based on the set environment, so as to complete the training of the deep Q network in the stage, wherein the method for performing one inner loop is specifically as follows: A1, updating the greedy policy step by step values, selecting an action according to the greedy policy, performing the action to obtain a reward, a new state, and a termination signal; A2, a transition tuple containing a current state, an action, a reward, a new state and a termination signal is stored in an experience replay buffer, and in response to the stored data in the experience replay buffer being greater than a preset number, a batch size of experience values is selected for one learning update; A3, whether the end point is reached is judged, if yes, the best reward, the path and the cumulative dose record are updated, one inner loop is completed, and if not, the time step is increased by one, and A1 is returned; S32, in response to the training of all curriculum stage deep Q networks being completed, a trained deep Q network model is output.

5. The method of claim 4, wherein, In A1, the greedy strategy The values are controlled by a cosine decay scheduler, time step t The expression for The expression for wherein is the minimum value of is the maximum value of is the minimum value of is the maximum value of is the number of time steps of the warm-up period, is the total number of time steps.

6. The method of claim 4, wherein, In A2, the method for selecting a batch size of experience values for one learning update is specifically as follows: B1. Prior sample batch size empirical value from experience replay buffer, calculate current Q value and target Q value y ; In the formula, r As a reward, As a discount factor for rewards, In order to take action The new state that is obtained later In the new state The maximum expected return that can be obtained by taking different actions. The parameters of the target Q-network, This is a termination signal; B2. According to the current Q value and the target Q value y Calculate the timing difference error , and then build the loss function ; wherein is a parameter of the current Q network, is the expected computation from the sampled experience batch from the priority replay buffer, is the importance sampling weight, B is the batch size; B3, the deep Q network is optimized by back propagation.

7. The method of claim 1, wherein, In S4, the method for searching for a minimum radiation dose path on a map by using the trained deep Q network model is specifically as follows: Based on the trained deep Q network model and the reward function, an inference search is performed on a grid map of a given start point-end point pair, a greedy strategy is adopted in the search process to gradually decide actions, the movement of an agent from a start point to an end point is simulated, and the cumulative radiation exposure and the path length are minimized to obtain an optimal path. 8.The radiation-aware path planning method based on deep Q network and curriculum learning according to claim 1, wherein, In S4, the method for performing transfer learning is specifically as follows: Based on the trained deep Q network model parameters and an environment modeling framework, the trained deep Q network model is loaded to a new agent, the fine tuning of the deep Q network model is performed, the source domain knowledge is inherited, and the loss of a new domain is minimized to obtain a fine-tuned deep Q network model.

Images