Orthodontic path planning method based on deep reinforcement learning

Abstract

The invention discloses an orthodontic path planning method based on deep reinforcement learning, and the method comprises the steps: modeling an orthodontic path planning problem as a Markov Decision Process (MDP), and solving the MDP by adopting an improved Deep Description Policy Grade (DDPG) algorithm. According to the technical scheme, the method comprises the following steps: processing a high-dimensional environment state (including information such as current and target postures of teeth) and an action (translating or rotating the teeth) space by adopting an Actor-Critic structure based on a Transform, and modeling a complex relationship between the teeth; a dynamic action mask mechanism is introduced into the Actor network to restrain the number of teeth moving in a single step so as to generate sparse actions meeting the clinical orthodontic law; and a two-stage curriculum learning strategy is adopted to realize training process optimization from loose constraint to strict constraint. According to the method, the safe, smooth and efficient tooth movement path can be automatically generated, and the intelligent level of clinical orthodontic planning can be improved.

Description

Technical Field

[0001] This invention relates to the fields of digital orthodontics and artificial intelligence, specifically to a method for planning orthodontic pathways based on deep reinforcement learning. Background Technology

[0002] Orthodontics plays a vital role in improving tooth alignment and occlusion by correcting misalignment, crowding, or gaps, thereby optimizing oral functions such as chewing, speech, and breathing, and enhancing the aesthetics of teeth and face. Orthodontic pathway planning, which designs a safe (e.g., minimal collision between adjacent teeth) and efficient (e.g., minimal tooth movement) path from the current pathological state to the target alignment, is the foundation of orthodontic treatment. Currently, orthodontic pathway planning in clinical practice is primarily done manually by orthodontists. However, this process is not only time-consuming but also highly dependent on the doctor's experience, making it difficult to meet the demands of modern orthodontics for high-efficiency, high-quality treatment.

[0003] Existing automation technologies mainly fall into two categories. One is optimization-based methods, which treat orthodontic path planning as an optimization problem, designing an objective function that incorporates clinical orthodontic knowledge and solving it using algorithms such as genetic algorithms or particle swarm optimization. However, these methods are typically computationally expensive, prone to getting trapped in local optima, and struggle to accurately model complex clinical rules. The other category is data-driven supervised learning methods, which primarily utilize deep neural networks to automatically learn patterns for orthodontic paths from large amounts of expert data. While this method avoids explicit modeling of complex orthodontic knowledge, its performance heavily depends on the quality and diversity of the training data. Due to patient privacy protection and high collection costs, high-quality orthodontic path data is currently very limited, leading to performance bottlenecks for data-driven methods. Therefore, there is an urgent need for a new method that can overcome the challenges of modeling clinical orthodontic rules and the limited availability of high-quality expert data, automatically generating safe and efficient orthodontic paths. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for orthodontic path planning based on deep reinforcement learning. This method combines the learning ability of data-driven methods with the exploratory ability of optimization methods, aiming to maximize the long-term cumulative reward driven by important clinical orthodontic rules. It uses a deep neural network-based agent to interact with the environment, which contains information such as the current and target states of the teeth, and learns from expert data to construct a strategy for generating optimal tooth movements.

[0005] This invention is achieved through the following technical solution: A method for orthodontic path planning based on deep reinforcement learning is proposed. The orthodontic path planning problem is modeled as a Markov Decision Process (MDP), and then the DDPG algorithm is used to solve the MDP. The method includes the following steps: Step 1: Obtain the patient's digital dental arch model data, including the initial tooth posture, the target tooth posture, and its 3D tooth model; Step 2: Input the patient's 3D tooth model into a pre-trained geometric encoder to generate a static shape feature code for each tooth; Step 3: For the current orthodontic planning time step, construct a state matrix to characterize the current orthodontic environment using the current pose, target pose, pose difference, previous action, and shape feature encoding of all teeth; Step 4: Input the state matrix into the Actor network of the DDPG algorithm. The Actor network outputs a basic action matrix and a mask vector in parallel. Apply the mask vector to the basic action matrix to generate a sparse action matrix that defines part of the tooth movement. Step 5: Apply the sparse action matrix to the current state matrix and calculate the state matrix for the next time step; Step 6: Based on the changes before and after the state update, and the predefined reward function, calculate a reward value to evaluate the quality of the sparse action matrix; Step 7: Store the self-exploration experience tuple consisting of the current state matrix, action matrix, reward value and new state matrix into an experience replay buffer, and sample data from the buffer to update the parameters of the Actor network and Critic network in the DDPG algorithm by maximizing the long-term cumulative reward. Step 8: Repeat steps 3 to 7 to train the Actor network and Critic network in the DDPG algorithm; Step 9: Input a new sample containing the initial pose, target pose, and 3D model of the teeth. Using the Actor network trained in Step 8, generate orthodontic path points step by step through autoregression until the difference between the current pose and the target pose of the teeth is less than a preset threshold, thus completing the orthodontic path planning.

[0006] Furthermore, step (1) specifically includes the following sub-steps: (1.1) Initial tooth posture: The tooth arrangement of the patient before orthodontics. The posture of each tooth is defined by its three-dimensional position vector and six-dimensional rotation vector in the reference coordinate system. (1.2) Target tooth posture: The aligned tooth arrangement designed by the orthodontist; (1.3) Three-dimensional tooth model: a triangular mesh model of each tooth.

[0007] Furthermore, step (2) specifically includes the following sub-steps: (2.1) A pre-trained geometric encoder is used to extract the shape features of each tooth; (2.2) For each tooth, a point cloud is uniformly sampled from its three-dimensional mesh and input into the encoder. The encoder outputs a 108-dimensional shape feature code that remains unchanged throughout the path planning process. (2.3) The shape feature encoding obtained in step (2.2) is pre-trained through a self-supervised point cloud completion task. The training objective is to enable the network to reconstruct a complete point cloud given a fragmented point cloud. The loss function is to minimize the chamfer distance between the reconstructed point cloud and the original complete point cloud. After the pre-training is completed, the parameters of the encoder are frozen.

[0008] Furthermore, step (3) specifically involves: at each time step of orthodontic path planning, a state matrix that comprehensively describes the current environment needs to be constructed; where each row is a state vector of a tooth, consisting of the following five parts: Current pose: The tooth's current 3D position vector and 6D rotation vector; Target pose: The final pose expected by the teeth, namely the three-dimensional position vector and the six-dimensional rotation vector, which provides the agent with action direction information; Attitude difference: The difference between the current attitude and the target attitude, providing the agent with an explicit error signal; Previous action: The action taken at the previous time step, including the increment of the three-dimensional position vector and the increment of the six-dimensional rotation vector, used to guide the agent to learn and generate a more coherent and smoother sequence of actions; Shape encoding: The feature vector obtained from step (2) provides the agent with the shape information of the teeth; At the start of orthodontic path planning, the initial pose is set to the current pose, and the vector representing the action of the previous time step is initialized to a zero vector, indicating that there is no preceding action in the initial state.

[0009] Specifically, the Actor network in step (4) contains two parallel output heads: an action head and a mask head; the action head outputs a basic action matrix, while the mask head outputs a mask vector with dimensions corresponding to the number of teeth and element values ​​between 0 and 1.

[0010] Specifically, the reward value in step (6) is obtained by weighted summation of four parts: progress reward for the tooth moving toward the target posture, collision penalty for collision between adjacent teeth, smoothness penalty for discontinuous tooth movement, and terminal reward given when the tooth successfully reaches the target posture.

[0011] Specifically, the DDPG algorithm in step (8) is a reinforcement learning method based on an Actor-Critic structure. Both the Actor network and the Critic network adopt the Transformer architecture. The Transformer architecture uses a multi-head self-attention mechanism to interactively model the features of different teeth in the state matrix to capture the interdependencies between teeth. The Actor network represents the policy function. Used in a given state The optimal action is generated using a Critic network, which represents the action-value function and is used to evaluate actions. Good or bad.

[0012] Specifically, the multi-head self-attention mechanism adds a learnable bias matrix to the standard attention score calculation. Its calculation expression is: Each element of the bias matrix , Teeth With teeth The index distance between them These are learnable parameters; These are query, key, and value matrices, respectively. For the size of the feature dimension, It is a learnable relative position offset matrix; The Actor network minimizes the policy loss function. The policy loss function is updated by a policy gradient term. and a mask regularization term Composition, represented as: ,in The aim is to constrain the Actor network so that its outputs can obtain higher evaluation values ​​for actions. The aim is to constrain the sparsity and binary nature of the mask vector output by the Actor network; the mask regularization term The mathematical expression is: The first item is for punishment. Exceeding the preset threshold The first term is an entropy regularization term that penalizes mask values ​​that deviate from 0 or 1, thus encouraging sparsity; the second term is an entropy regularization term that penalizes mask values ​​that deviate from 0 or 1, thus encouraging binarization. and These are the corresponding weighting coefficients; The Critic network minimizes the temporal difference error. The update is performed, and its mathematical expression is: The time-series difference target value It is obtained by averaging the 1-step temporal difference objective and the 3-step temporal difference objective.

[0013] Further, in step (8), the Actor network and Critic network in the DDPG algorithm are trained; the Actor network and Critic network are trained using a two-stage learning method, which includes the following sub-steps: (8.1) First-stage training: Training is conducted in an environment with a first set of preset hyperparameters, which correspond to relaxed clinical orthodontic constraints; the first set of preset hyperparameters includes low collision penalty coefficient, low path discontinuity penalty, and ; (8.2) Second-stage training: Using the policy network obtained in the first-stage training as initialization, further training is performed in a simulation environment with a second set of preset hyperparameters. The second set of hyperparameters corresponds to strict clinical orthodontic constraints; the second set of preset hyperparameters includes a high collision penalty coefficient, a high path discontinuity penalty, and and .

[0014] Specifically, an experience replay buffer is used during the training of the Actor network and the Critic network. Before training begins, the experience replay buffer is pre-filled with expert experience tuples consisting of expert orthodontic path data designed by orthodontists. At the same time, the self-exploration experience tuples in step seven are also stored in this buffer. The experience replay buffer is a priority experience replay (PER) buffer, and when sampling data from this buffer for network updates, the expert experience tuples consisting of expert orthodontic path data are given a higher sampling priority.

[0015] The beneficial effects of this invention are as follows: This invention combines the learning capabilities of data-driven methods with the exploratory capabilities of optimization methods to construct a strategy for generating optimal tooth movement, aiming to maximize the long-term cumulative rewards driven by important clinical orthodontic rules. This strategy effectively utilizes expert data while accurately modeling key clinical orthodontic rules. Compared to existing automated methods, the method described in this invention generates safer, smoother, and more efficient orthodontic pathways. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the method described in this invention; Figure 2 This is a schematic diagram of the two-stage course training in the method described in this invention. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0018] This invention proposes a method for orthodontic path planning based on deep reinforcement learning. For example... Figure 1 As shown, its core principle lies in modeling the orthodontic path planning problem as a Markov Decision Process (MDP) and solving it using an improved Deep Deterministic Policy Gradient (DDPG) algorithm. The main steps include: Step 1: Data Acquisition.

[0019] This method first acquires the patient's digital dental arch model data, specifically including: Initial tooth orientation: The alignment of the patient's teeth before orthodontic treatment. The orientation of each tooth is defined by its three-dimensional position vector and six-dimensional rotation vector in the reference coordinate system.

[0020] Target tooth orientation: The aligned tooth arrangement designed by an orthodontist.

[0021] 3D tooth model: a triangular mesh model of each tooth.

[0022] Step 2: Extract tooth shape feature codes.

[0023] A pre-trained geometric encoder is used to extract the shape features of each tooth. For each tooth, a point cloud is uniformly sampled from its 3D mesh and input into the encoder. The encoder outputs a 108-dimensional shape feature encoding that remains invariant throughout the path planning process. The encoder is pre-trained using a self-supervised point cloud completion task. The training objective is for the network to reconstruct a complete point cloud given a fragmented point cloud. The loss function minimizes the chamfer distance between the reconstructed point cloud and the original complete point cloud. After pre-training, the encoder's parameters are frozen.

[0024] Step 3: Dynamic construction of the environment state matrix.

[0025] At each time step of orthodontic path planning All of these require the construction of a state matrix that can comprehensively describe the current environment. The state matrix has a dimension of 28×144, where each row is the state vector of a tooth, consisting of the following five parts: Current posture ( ): The current three-dimensional position vector of the tooth and six-dimensional rotation vector , Index representing teeth ( 1,...,28, corresponding to the 28 teeth in the patient's mouth excluding wisdom teeth).

[0026] Target attitude ( ): The expected final pose of the teeth (3D position vector) and six-dimensional rotation vector This provides the agent with information on the direction of its actions.

[0027] poor posture ( The difference between the current pose and the target pose provides the agent with an explicit error signal.

[0028] Previous action ( ): Actions taken at the previous time step, including the increment of the 3D position vector. and the increment of the six-dimensional rotation vector It is used to guide agents to learn and generate more coherent and smoother action sequences.

[0029] Shape coding ( The feature vector obtained from step two provides the agent with tooth shape information.

[0030] At the start of orthodontic path planning (time step) The initial pose is set as the current pose, and the vector representing the action of the previous time step is... It is initialized as a vector of all zeros, indicating that there are no preceding actions in the initial state.

[0031] Step 4: Sparse action generation based on dynamic mask.

[0032] The environmental state matrix Input the Actor network of the DDPG algorithm to generate a sparse action matrix. .

[0033] The output of the Actor network is connected in parallel to two independent multilayer sensing heads: one action head (outputting the basic action matrix). ) and a masking terminal (output mask vector) ). Mask vector Acting on the basic action matrix The final sparse action matrix is ​​obtained. .

[0034] Step 5: Environmental Interaction.

[0035] The sparse action matrix Current state matrix applied to the teeth Calculate the new state matrix for the next time step. .

[0036] Step 6: Calculate the reward function.

[0037] Calculate a reward value after each environmental interaction. This is used to guide the learning of intelligent agents. Specifically, This includes progress rewards for teeth moving toward the target posture. Collision penalty for collisions between adjacent teeth ), resulting in a smoothness penalty due to discontinuous tooth movement ( ) and the terminal reward given when the teeth successfully reach the target posture ( ).

[0038] Furthermore, the progress reward for teeth moving towards the target pose. include and .for First, it is necessary to calculate the square of the Euclidean distance between the pose of each tooth before and after the environmental interaction and its desired target pose, such as... and , then calculate ,in =16 (distance scaling factor) =28 (representing the number of teeth). Similarly, ,in , and These represent the squares of the angle difference between the teeth and the target pose before and after environmental interaction.

[0039] Collision penalty for collisions between adjacent teeth ( Used to evaluate the execution action matrix The newly generated state matrix Safety is ensured. Specifically, the GJK algorithm is used to detect collisions between all adjacent teeth. If a collision occurs, its penetration depth is calculated. and receive a collision penalty. Collision penalty The calculation formula is: The final collision penalty applies to all... Summing up the detection pairs that could potentially collide (a detection pair is defined as two adjacent teeth in space). .

[0040] Smoothness penalty occurs due to discontinuous tooth movement. Used to evaluate the current action matrix Action matrix relative to the previous time step The degree of drastic change. The calculation formula is: ,in This is a hyperparameter; adjust its value accordingly. The number of teeth.

[0041] The terminal reward given when the teeth successfully reach the target posture ( This is used to provide a clear and significant positive feedback when a task is successfully completed. Specifically, if an action matrix is ​​executed... The generated new state matrix If for any tooth, the Euclidean distance between it and the target orientation does not exceed 0.2 mm and the angular difference does not exceed 2°, then a condition is given. Positive rewards, otherwise .

[0042] Step 7: Experience review and strategy optimization.

[0043] The current environment state matrix will be used. Action matrix Reward Value and the new state matrix The self-exploratory experience tuples are stored in an experience replay buffer, and data is sampled from this buffer. The weights of the Actor and Critic networks in the DDPG algorithm are updated by maximizing long-term cumulative rewards. The experience replay buffer has a capacity of 1 million and is pre-filled with trajectory data from 9,000 expert cases before training begins. During sampling, this expert data is given the highest sampling priority through a Prioritized Experience Replay (PER) mechanism.

[0044] Step 8: Repeat steps 3 to 7 to train the Actor network and Critic network of the DDPG algorithm.

[0045] The DDPG algorithm comprises an Actor network, a Critic network, and their corresponding target networks. The target network primarily enhances training stability and smooths the training process. It updates parameters by slowly replicating the parameters of the main network. All training data during the training process originates from the experience replay buffer described in step seven.

[0046] The Actor network loss It consists of two parts: Policy gradient term: The aim is to maximize the evaluation value given by the Critic network.

[0047] Sparsity constraints: , intended to punish Exceeding the preset threshold In this situation, at the same time encourage Binarization.

[0048] The Critic network minimizes the temporal difference error. The update is performed, and its mathematical expression is: To balance bias and variance, its learning objective is... It is the average of the 1-step time-difference objective and the 3-step time-difference objective.

[0049] Furthermore, to better model the spatial relationships of teeth, this invention introduces a learnable relative position encoding scheme based on tooth index distance. This scheme adds a learnable bias term to the standard attention score calculation. The calculation formula is: Among them, the bias term , Teeth With teeth The index distance between them These are learnable parameters. This mechanism adjusts the effect of inter-tooth index distance on attention scores, prompting the model to generate higher attention scores between two spatially closer teeth.

[0050] In addition, such as Figure 2 As shown, the entire training process can be divided into two stages: Phase 1 (0 steps - 2 million steps): Coarse-grained exploration, aiming to quickly find a path from the initial pose to the target pose. This phase uses a high learning rate (Critic: 1e-3, Actor: 1e-4, employing a linear annealing strategy), high exploration noise, and a lenient penalty (…). 10 ( ), ) and relaxed sparsity constraints ( and This encourages agents to quickly find a path from the initial pose to the target pose.

[0051] Phase Two (2 million to 4 million steps): Fine-grained optimization, aiming to improve path quality. This phase reduces the learning rate and exploration noise, and imposes strict penalties. ) and strict sparsity constraints ( and This is to force the generation of sparse orthodontic planning paths that are in line with clinical practice.

[0052] Step 9: Iterative path generation.

[0053] Input a new sample (containing the initial pose, target pose, and 3D model of the teeth), and use a trained Actor network to generate orthodontic path points step by step through autoregression until the difference between the current pose and the target pose of the teeth is less than a preset threshold.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0055] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for orthodontic path planning based on deep reinforcement learning, characterized in that, The orthodontic pathway planning problem is modeled as a Markov Decision Process (MDP), and then the DDPG algorithm is used to solve the MDP. The method includes the following steps: Step 1: Obtain the patient's digital dental arch model data, including the initial tooth posture, the target tooth posture, and its 3D tooth model; Step 2: Input the patient's 3D tooth model into a pre-trained geometric encoder to generate a static shape feature code for each tooth; Step 3: For the current orthodontic planning time step, construct a state matrix to characterize the current orthodontic environment using the current pose, target pose, pose difference, previous action, and shape feature encoding of all teeth; Step 4: Input the state matrix into the Actor network of the DDPG algorithm. The Actor network outputs a basic action matrix and a mask vector in parallel. Apply the mask vector to the basic action matrix to generate a sparse action matrix that defines part of the tooth movement. Step 5: Apply the sparse action matrix to the current state matrix and calculate the state matrix for the next time step; Step 6: Based on the changes before and after the state update, and the predefined reward function, calculate a reward value to evaluate the quality of the sparse action matrix; Step 7: Store the self-exploration experience tuple consisting of the current state matrix, action matrix, reward value and new state matrix into an experience replay buffer, and sample data from the buffer to update the parameters of the Actor network and Critic network in the DDPG algorithm by maximizing the long-term cumulative reward. Step 8: Repeat steps 3 to 7 to train the Actor network and Critic network in the DDPG algorithm; Step 9: Input a new sample containing the initial pose, target pose, and 3D model of the teeth. Using the Actor network trained in Step 8, generate orthodontic path points step by step through autoregression until the difference between the current pose and the target pose of the teeth is less than a preset threshold, thus completing the orthodontic path planning.

2. The method for orthodontic path planning based on deep reinforcement learning according to claim 1, characterized in that, Step (1) specifically includes the following sub-steps: (1.1) Initial tooth posture: The tooth arrangement of the patient before orthodontics. The posture of each tooth is defined by its three-dimensional position vector and six-dimensional rotation vector in the reference coordinate system. (1.2) Target tooth posture: The aligned tooth arrangement designed by the orthodontist; (1.3) Three-dimensional tooth model: a triangular mesh model of each tooth.

3. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, Step (2) specifically includes the following sub-steps: (2.1) A pre-trained geometric encoder is used to extract the shape features of each tooth; (2.2) For each tooth, a point cloud is uniformly sampled from its three-dimensional mesh and input into the encoder. The encoder outputs a 108-dimensional shape feature code that remains unchanged throughout the path planning process. (2.3) The shape feature encoding obtained in step (2.2) is pre-trained through a self-supervised point cloud completion task. The training objective is to enable the network to reconstruct a complete point cloud given a fragmented point cloud. Its loss function is to minimize the chamfer distance between the reconstructed point cloud and the original complete point cloud; after pre-training, the parameters of the encoder are frozen.

4. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, Step (3) specifically involves constructing a state matrix that comprehensively describes the current environment at each time step of orthodontic path planning; each row is a state vector of a tooth, consisting of the following five parts: Current pose: The tooth's current 3D position vector and 6D rotation vector; Target pose: The final pose expected by the teeth, namely the three-dimensional position vector and the six-dimensional rotation vector, which provides the agent with action direction information; Attitude difference: The difference between the current attitude and the target attitude, providing the agent with an explicit error signal; Previous action: The action taken at the previous time step, including the increment of the three-dimensional position vector and the increment of the six-dimensional rotation vector, used to guide the agent to learn and generate a more coherent and smoother sequence of actions; Shape encoding: The feature vector obtained from step (2) provides the agent with the shape information of the teeth; At the start of orthodontic path planning, the initial pose is set to the current pose, and the vector representing the action of the previous time step is initialized to a zero vector, indicating that there is no preceding action in the initial state.

5. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, The Actor network in step (4) contains two parallel output heads: an action head and a mask head; the action head outputs a basic action matrix, while the mask head outputs a mask vector with dimensions corresponding to the number of teeth and element values ​​between 0 and 1.

6. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, The reward value in step (6) is obtained by weighted summation of four parts: progress reward for the tooth moving toward the target posture, collision penalty for collision between adjacent teeth, smoothness penalty for discontinuous tooth movement, and terminal reward given when the tooth successfully reaches the target posture.

7. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, The DDPG algorithm in step (8) is a reinforcement learning method based on the Actor-Critic structure. Both the Actor network and the Critic network adopt the Transformer architecture. The Transformer architecture uses a multi-head self-attention mechanism to interactively model the features of different teeth in the state matrix in order to capture the interdependencies between teeth. Where the Actor network represents the policy function. Used in a given state The optimal action is generated using a Critic network, which represents the action-value function and is used to evaluate actions. Good or bad.

8. The orthodontic path planning method based on deep reinforcement learning according to claim 7, characterized in that, The multi-head self-attention mechanism adds a learnable bias matrix to the standard attention score calculation. Its calculation expression is: Each element of the bias matrix , Teeth With teeth The index distance between them These are learnable parameters; These are query, key, and value matrices, respectively. For the size of the feature dimension, It is a learnable relative position offset matrix; The Actor network minimizes the policy loss function. The policy loss function is updated by a policy gradient term. and a mask regularization term Composition, represented as: ,in The aim is to constrain the Actor network so that its outputs can obtain higher evaluation values ​​for actions. The aim is to constrain the sparsity and binary nature of the mask vector output by the Actor network; the mask regularization term The mathematical expression is: The first item is for punishment. Exceeding the preset threshold The first term is an entropy regularization term that penalizes mask values ​​that deviate from 0 or 1, thus encouraging sparsity; the second term is an entropy regularization term that penalizes mask values ​​that deviate from 0 or 1, thus encouraging binarization. and These are the corresponding weighting coefficients; The Critic network minimizes the temporal difference error. The update is performed, and its mathematical expression is: The time-series difference target value It is obtained by averaging the 1-step temporal difference objective and the 3-step temporal difference objective.

9. The orthodontic path planning method based on deep reinforcement learning according to claim 1, characterized in that, In step (8), the Actor network and Critic network in the DDPG algorithm are trained; Its Actor and Critic networks are trained using a two-stage learning method, which includes the following sub-steps: (8.1) First-stage training: Training is conducted in an environment with a first set of preset hyperparameters, which correspond to relaxed clinical orthodontic constraints; the first set of preset hyperparameters includes low collision penalty coefficient, low path discontinuity penalty, and ; (8.2) Second-stage training: Using the policy network obtained in the first-stage training as initialization, further training is performed in a simulation environment with a second set of preset hyperparameters. The second set of hyperparameters corresponds to strict clinical orthodontic constraints; the second set of preset hyperparameters includes a high collision penalty coefficient, a high path discontinuity penalty, and and .

10. The method according to claim 1, characterized in that, During the training of the Actor and Critic networks, an experience replay buffer is used. Before training begins, the experience replay buffer is pre-filled with expert experience tuples consisting of expert orthodontic path data designed by orthodontists. At the same time, the self-exploration experience tuples in step seven are also stored in this buffer. The experience replay buffer is a priority experience replay (PER) buffer, and when sampling data from this buffer for network updates, the expert experience tuples consisting of expert orthodontic path data are given a higher sampling priority.

Images