Neurosurgery robot obstacle avoidance method based on reinforcement learning

Abstract

The application relates to the technical field of robot control, in particular to a neurosurgery robot obstacle avoidance method based on reinforcement learning, which comprises the following steps: determining a target space based on a three-dimensional environment image, determining state space information of a neuroendoscope, and determining action space information of the neuroendoscope; determining a reward model and a punishment model of a surgical robot movement action; training by using a deep deterministic policy gradient algorithm to obtain an obstacle avoidance control model; running a neurosurgery robot, acquiring real-time states by using a position sensor and a force sensor, inputting the obstacle avoidance control model, and obtaining a control action; and controlling the surgical robot by using the control action to complete movement obstacle avoidance of the surgical robot; and the application can improve the obstacle avoidance precision, real-time performance and adaptability of the neurosurgery robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and particularly relates to a neurosurgical operation robot obstacle avoidance method based on reinforcement learning. BACKGROUND

[0002] The neuroendoscopic robot is an advanced medical auxiliary technology, which can realize precise positioning and flexible operation in a narrow physiological space through a high-precision tracking positioner and a precise mechanical arm as the operator's "eyes" and "hands". The most significant advantage of the neuroendoscopic robot is that it can determine the three-dimensional anatomical structure with the aid of image information and realize precise control of the mechanical arm through force feedback technology. It can clearly identify the spatial position of the key tissues and automatically adjust the operation force and operation direction through intelligent control strategy, thereby maximizing the protection of the surrounding tissues. Compared with the traditional method, this intelligent auxiliary system has the characteristics of clear vision, small trauma, high accuracy, etc., and has become an important technical innovation in the medical field.

[0003] The existing intelligent navigation obstacle avoidance technology mainly includes two categories: space navigation technology based on preset images and robot auxiliary technology based on real-time force feedback. Among them, the navigation technology based on preset preoperative images establishes a spatial positioning model through pre-acquired three-dimensional image data (such as CT or MRI scans), and the system uses image registration and segmentation methods to construct the spatial relationship graph of each target object for real-time guidance of device movement. However, due to the deformation of soft tissues, the change of target object position, and the deviation of the device during movement, these factors will cause the deviation of the preset spatial model from the actual situation, affecting the positioning accuracy. The robot auxiliary technology based on force feedback is equipped with a force sensor at the end of the mechanical arm, which detects the contact force for obstacle avoidance control, but also has many problems: the force signal needs to be transmitted to the sensor through multiple mechanical structures, resulting in insufficient real-time performance, the force from various directions in the complex structure will interfere with each other, causing measurement error to be large, and the control force applied by the system to the operator is sensitive, making the adaptive ability limited. In the rapidly changing working environment, the existing system is difficult to make timely and accurate obstacle avoidance decisions. In summary, the existing intelligent navigation obstacle avoidance technology still needs to be improved in terms of accuracy, real-time performance and adaptive ability. SUMMARY

[0004] In view of the above problems, the present application provides a neurosurgical operation robot obstacle avoidance method based on reinforcement learning, which solves the problem of poor precision, real-time performance and adaptability of the surgical robot obstacle avoidance in the prior art.

[0005] The application provides a neural surgery robot obstacle avoidance method based on reinforcement learning, the neural surgery robot comprises a neuroendoscope, a mechanical arm, a position sensor and a force sensor, the mechanical arm is provided with a plurality of robot joints, and the neuroendoscope is connected to one end of the mechanical arm, and the method is characterized in that the method comprises the following steps:

[0006] Step S1, determining a target space based on a three-dimensional environment image, determining state space information of the neuroendoscope, and determining action space information of the neuroendoscope;

[0007] Step S2, determining a reward model of a movement action of the surgery robot based on a distance between the position of the neuroendoscope and a motion end point in the target space, and determining a punishment model of the movement action of the surgery robot based on a distance between the neuroendoscope and a key environment structure in the target space and a contact force between the neuroendoscope and a contactable structure in the target space;

[0008] Step S3, training by using a deep deterministic policy gradient algorithm based on the state space information and the action space information and based on the reward model and the punishment model of the movement action of the surgery robot, to obtain an obstacle avoidance control model;

[0009] Step S4, running the neural surgery robot, acquiring real-time states by using the position sensor and the force sensor, inputting the obstacle avoidance control model, and obtaining a control action;

[0010] Step S5, controlling the surgery robot by using the control action, and completing movement obstacle avoidance of the surgery robot.

[0011] Preferably, the step S1 specifically comprises:

[0012] Step S1-1, acquiring a tomographic image of an area to be analyzed by using a medical imaging device, performing three-dimensional reconstruction on the tomographic image to acquire a three-dimensional model, and determining a target space based on the three-dimensional model, wherein the target space is a movement space of the surgery robot;

[0013] Step S1-2, determining a position state and a force state of the neuroendoscope as the state space information;

[0014] Step S1-3, determining a movement state of the mechanical arm as the action space information.

[0015] Preferably, the state space information in the step S1-2 specifically comprises:

[0016] a position P of the neuroendoscope e a position P of a key environment structure a a contact force F between the neuroendoscope and a contactable structure axis and a position P of a motion end point t .

[0017] The action space information in step S1-3 specifically includes: rotation angle increments [Δθ1, Δθ 2, , Δθ 3, , Δθ 4, , Δθ5, Δθ6] of the 6 joints of the mechanical arm 6.

[0018] Preferably, the step S2 specifically includes:

[0019] Step S2-1, determining a reaching reward, the closer the distance between the neuroendoscope and the motion end point, the greater the reaching reward;

[0020] Step S2-2, determining a collision penalty based on the distance between the neuroendoscope and the key environmental structure and the size relationship between the safe distance and the alert distance;

[0021] Step S2-3, determining a contact force penalty based on the contact force between the neuroendoscope and the contactable structure in the target space and the size relationship between the first and second contact force thresholds.

[0022] Preferably, in step S2-1, the calculation expression of the reaching reward is:

[0023]

[0024] Wherein, R1 represents the reaching reward, Dis(P e , P t ) represents the distance between the neuroendoscope position P e and the motion end point position P t , and a represents the reaching reward normalization coefficient;

[0025] In step S2-2, the calculation expression of the collision penalty is:

[0026]

[0027] Wherein, R2 represents the collision penalty, Dis(P e , P a ) represents the distance between the neuroendoscope position P e and the key environmental structure position P a , L1 represents the alert distance, L2 represents the safe distance, and β represents the collision penalty normalization coefficient;

[0028] In step S2-3, the calculation expression of the contact force penalty is:

[0029]

[0030] Wherein, R3 represents the contact force penalty, F axisrepresents the contact force of the neuroendoscope with the contactable structure, T1 and T2 represent the first and second contact force thresholds respectively, and γ is a contact force penalty normalization coefficient, is a first constant.

[0031] Preferably, the step S3 specifically comprises:

[0032] The state space and action space of the neuroendoscope are taken as the state space and action space of the DDPG algorithm, and the reward and penalty of the surgical robot motion action are taken as the reward and penalty of the DDPG algorithm, so that the reinforcement learning training is performed, and finally the obstacle avoidance control model is obtained.

[0033] In the DDPG algorithm reinforcement learning training process, the action a t is applied to the current time state s t , the next time state s t+1 is obtained, the position of the neuroendoscope in the next time state s t+1 , the position of the key environmental structure, the contact force of the neuroendoscope with the contactable structure and the motion end position are determined, the on-site reward, collision penalty and contact force penalty of the current time are determined, and the current time reward is determined according to the on-site reward, collision penalty and contact force penalty of the current time.

[0034] Preferably, the specific calculation method of the current time reward determined by the on-site reward, collision penalty and contact force penalty of the current time is:

[0035] For any time, the following expression is used to calculate the reward r:

[0036] r = R1 + R2 + R3

[0037] Wherein, R1, R2, R3 are on-site reward, collision penalty and contact force penalty respectively.

[0038] Preferably, the step S4 specifically comprises:

[0039] The position sensor is used to collect the position of the neuroendoscope P e , the position of the key environmental structure P a and the motion end position P t in real time, and the force sensor is used to collect the contact force F axis of the neuroendoscope with the contactable structure in real time, as the real-time state.

[0040] The real-time state is input into the obstacle avoidance control model, and the corresponding control action is output, wherein the control action includes the rotation angle increment of each joint of the surgical robot mechanical arm.

[0041] Compared with the prior art, the present application has at least the following beneficial effects:

[0042] (1) The present application improves the accuracy and safety of the operation by using a neural surgery robot obstacle avoidance method based on reinforcement learning. The target space and the state and action space information of the neuroendoscope are determined by three-dimensional environment images, providing an accurate environment perception basis for subsequent obstacle avoidance control.

[0043] (2) The present application designs a reward model and a punishment model, and comprehensively considers the distance between the neuroendoscope and the target space and the contact force, to ensure that the robot can intelligently avoid key environmental structures during movement. The robot can continuously optimize its obstacle avoidance strategy during the operation, automatically learn the best obstacle avoidance path, and thus improve the obstacle avoidance efficiency and precision of the surgical robot.

[0044] (3) The present application trains and forms an obstacle avoidance control model through the deep deterministic policy gradient algorithm, and the robot can adjust the movement path in real time. Combined with the real-time data provided by the position sensor and the force sensor, the robot can dynamically respond to changes in the surgical environment and adjust the obstacle avoidance strategy in time. It can maintain the obstacle avoidance precision and adaptability of the surgical robot. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings are for the purpose of illustrating preferred embodiments of the present application and are not to be construed as limiting the application.

[0046] Figure 1 A neural surgery robot obstacle avoidance method based on reinforcement learning is provided for the present application.

[0047] Figure 2 A principle diagram of the deep deterministic policy gradient method provided for the present application.

[0048] Figure 3 An intracranial structure diagram constructed using medical images is provided for the present application.

[0049] Figure 4 A neuroendoscope mechanical arm model diagram is provided for the present application.

[0050] Figure 5 A neuroendoscope collision punishment diagram is provided for the present application.

[0051] Figure 6 A neuroendoscope contact force punishment diagram is provided for the present application. DETAILED DESCRIPTION

[0052] In order to enable the above-mentioned objects, features and advantages of the present application to be clearer, the present application will be described in further detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0053] In order to illustrate the effectiveness of the method of the present application, the above technical solutions of the present application will be described in detail below through a specific embodiment. One specific embodiment of the present application discloses a neural surgery robot obstacle avoidance method based on reinforcement learning, as shown in Figure 1 、 Figure 4 The neural surgery robot comprises a mechanical arm, the mechanical arm is connected with a neuroendoscope at the end, the mechanical arm has a plurality of robot joints, and the movement of the neuroendoscope is controlled by the mechanical arm. The neural surgery robot further comprises a position sensor, a force sensor and the like, and the specific implementation steps are as follows:

[0054] Step S1, determining a target space based on a three-dimensional environment image, determining state space information of the neuroendoscope, and determining action space information of the neuroendoscope.

[0055] In this step, a tomographic image of the region to be analyzed, i.e. a three-dimensional environment image, is obtained by a medical imaging device (such as MRI, CT, etc.), and the tomographic image is spatially registered and fused by a three-dimensional reconstruction algorithm to generate a three-dimensional model containing tubular structures, abnormal regions and key environmental structures, as shown in Figure 3 .

[0056] The target space is determined based on the three-dimensional model, and the target space is the space in which the surgical robot moves. In some embodiments, a three-dimensional coordinate system of the target space can be established, and the spatial coordinate positions of the tubular structures, abnormal regions and key structures are included.

[0057] The state space and the action space of the neuroendoscope in the subsequent reinforcement learning training process are defined.

[0058] The state space (State Space) represents all possible states of the neuroendoscope in the current environment, and a certain state in the state space includes the following data:

[0059] (1) Neuroendoscope position P e : representing the current spatial position of the neuroendoscope, which can be obtained by the position sensor.

[0060] (2) Key environmental structure position P aThe positions in the target space, such as a tubular structure, an abnormal area, and a key structure, can be acquired by the position sensor.

[0061] (3) Contact force P of the neuroendoscope and the contactable structure axis The feedback data indicating whether the neuroendoscope is in contact with the contactable structure and the contact force between the neuroendoscope and the contactable structure can be acquired by the force sensor.

[0062] (4) Position P of the motion end point t The position P of the motion end point represents the target position of the current motion of the neuroendoscope.

[0063] The action space represents the motion information of each joint of the mechanical arm of the neurosurgical robot. In the neuroendoscopic surgery assisted by the robot, the neuroendoscope is mounted on the end of the robot, and the mechanical arm controls all the motions of the neuroendoscope. In some embodiments, the mechanical arm has 6 joints, and the rotation angle increment a of each joint of the mechanical arm is determined as a certain action in the action space. 2, , Δθ 3, , Δθ 4, , Δθ5, Δθ6].

[0064] Through the above steps, the target space model, the state space of the neuroendoscope, and the action space of the neuroendoscope are established based on three-dimensional imaging for reinforcement learning training.

[0065] In step S2, a reward model of the motion action of the surgical robot is determined based on the distance between the position of the neuroendoscope and the motion end point in the target space, and a punishment model of the motion action of the surgical robot is determined based on the distance between the neuroendoscope and the key environmental structure in the target space and the contact force between the neuroendoscope and the contactable structure in the target space.

[0066] In reinforcement learning, the reward is a key factor in determining how the model learns, and a good reward design can guide the system to learn to make appropriate decisions in a complex environment.

[0067] In the scene of controlling the obstacle avoidance motion of the neurosurgical robot, the present application designs the calculation method of reward and punishment according to the complex situation in the scene, including:

[0068] (1) On-site reward: the training target is to send the neuroendoscope to the motion end point position, so the closer to the motion end point position, the greater the reward, and after reaching the position, the reward is 1, indicating that the task is completed.

[0069] The expression of the on-site reward is:

[0070]

[0071] wherein R1 represents a reward for reaching the goal, Dis(P e , P t ) represents a distance between the neuroendoscope position P e and the goal position P t , and a represents a normalization coefficient for the reward for reaching the goal.

[0072] (2) Collision penalty: In the process of neuroendoscope movement, it is necessary to ensure that no key environmental structure is touched. In order to further improve safety, a warning distance L1 is set. When the distance is less than L1, it is considered to be in a dangerous state, and the task fails. A safe distance L2 is further set. When the distance is greater than L2, it is considered that there is no collision risk. At this time, the penalty is set to 0, which means that it does not participate in feedback, and the training efficiency can be further improved. Between L1 and L2, it is considered that a penalty is needed, which is expressed as a function of the distance, as shown in Figure 5 .

[0073] The expression of the collision penalty is:

[0074]

[0075] wherein R2 represents a collision penalty, Dis(P e , P a ) represents a distance between the neuroendoscope position P e and the key environmental structure position P a , L1 represents a warning distance, L2 represents a safe distance, and β represents a normalization coefficient for the collision penalty.

[0076] (3) Contact force penalty: In the process of movement of the endoscope, contact with the contactable structure will exist. In order to ensure safety, when the contact force is greater than T2, it is considered to be dangerous, and the task fails. If the contact force is less than T1, it is considered to be safe, and does not participate in feedback. When the force is between T1 and T2, a penalty is needed, and the penalty base is increased, which considers that the contact force is more worthy of attention in the process of movement, as shown in Figure 6 .

[0077] The expression of the contact force penalty is:

[0078]

[0079] wherein R3 represents a contact force penalty, F axis represents a contact force between the neuroendoscope and the contactable structure, T1 and T2 respectively represent a first and a second contact force threshold, γ is a normalization coefficient for the contact force penalty, and is a first constant.

[0080] Step S3, according to the state space information and the action space information, and based on a reward model and a penalty model of the motion action of the surgical robot, a deep deterministic policy gradient algorithm is trained to obtain an obstacle avoidance control model;

[0081] The application performs reinforcement learning based on the state space information and the action space information and the reward and penalty of the motion action of the surgical robot.

[0082] The application adopts a deep deterministic policy gradient (DDPG) method, and the action space output by the method is a continuous deterministic action. The DDPG is implemented based on an Actor-Critic framework. An Actor network includes a main network μ(s|θ μ ) and a target network μ'(s|θ μ′ ), wherein θ μ and θ μ′ represent parameters of the main network and the target network respectively. The main network and the target network of the Actor network can take a current state s as input to obtain a corresponding action. A Critic network also includes a main network Q(s,a|θ Q ) and a target network Q'(s,a|θ Q′ ), wherein θ Q and θ Q′ represent parameters of the main network and the target network respectively. The main network and the target network of the Critic network can take a current state s and a current action a as input to obtain an evaluation value of the action a under the state s. A specific training process based on the DDPG is shown in Figure 2 .

[0083] Specifically in the application, the state space of the neuroendoscope and the action space of the neuroendoscope are taken as the state space and the action space of the DDPG algorithm, and the reward and the penalty of the motion action of the surgical robot are taken as the reward and the penalty of the DDPG algorithm, so as to perform reinforcement learning training, and finally obtain an obstacle avoidance control model.

[0084] A pseudo code of the specific training process based on the DDPG is as follows:

[0085]

[0086] In the above DDPG training process, the current time action a t includes a rotation angle increment of each joint of the mechanical arm at the current time, and the current time state s t includes a position of the neuroendoscope, a position of a key environmental structure, a contact force between the neuroendoscope and a contactable structure, and a motion end position at the current time.

[0087] In the DDPG training process, the action 'a' at the current moment is used. t Get the reward r at the current moment t The specific implementation method is as follows:

[0088] Action a at the current moment t Acting on the current state s t Get the state s of the next time step. t+1 According to the state s at the next moment t+1 The location of the neuroendoscopy, the location of key environmental structures, the contact force between the neuroendoscopy and accessible structures, and the location of the endpoint of the motion determine the current moment's arrival reward, collision penalty, and contact force penalty. The current moment's reward r is determined by these factors. t .

[0089] For any given time, the expression for calculating the reward r in this invention is:

[0090] r = R1 + R2 + R3

[0091] Through the training steps described above, this invention yields an obstacle avoidance control model for a neurosurgical robot. This model can take the robot's current state s as input and obtain the corresponding action.

[0092] Step S4: During the operation of the neurosurgical robot, the real-time status is obtained by the position sensor and force sensor, and input into the obstacle avoidance control model to obtain the control action.

[0093] In this step, during the actual operation of the neurosurgical robot, the position P of the neuroendoscope is collected in real time by the position sensor. e Key environmental structural location P a and the final position P of the movement t Simultaneously, the contact force F between the neuroendoscopy and the accessible structure is collected in real time using a force sensor. axis .

[0094] The collected position and force information is input into a pre-trained obstacle avoidance control model as state space information. Based on the current state space information, the obstacle avoidance control model calculates and outputs corresponding control actions to guide the surgical robot to adjust its motion trajectory and achieve intelligent obstacle avoidance during the movement of the surgical robot.

[0095] The control actions may include the rotation angle increments of each joint of the surgical robot arm: [Δθ1, Δθ 2, ,Δθ 3, ,Δθ 4, ,Δθ5,Δθ6).

[0096] Step S5, controlling the surgical robot by the control action to complete the motion obstacle avoidance of the surgical robot.

[0097] In this step, the instructions corresponding to the control action are transmitted to the drive modules of each mechanical arm of the neurosurgical surgical robot. The control action includes the rotation angle increment of each joint of the surgical robot mechanical arm, which specifically indicates the rotation direction and angle of each joint to ensure that the robot can move along the predetermined path and avoid obstacles in the environment in real time when performing the surgical task. Through precise joint angle adjustment, the surgical robot can smoothly and flexibly avoid possible collisions or interference, ensuring the smooth progress of the surgical process. In addition, the control system also dynamically adjusts the control strategy according to real-time sensor feedback data to cope with different changes in the surgical environment, ensuring that the robot always maintains the best motion trajectory and safety.

[0098] The detailed description of the application, although describing each action or step in a specific order, should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all the illustrated actions or steps to be performed to achieve the desired results. In certain circumstances, multi-tasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be combined in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately or in any suitable sub-combination. The above description is only the preferred specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed by the application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the application.

[0099] The above description is only the preferred specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed by the application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the application.

Claims

1. A neural surgery robot obstacle avoidance system based on reinforcement learning, the neural surgery robot comprising a neuroendoscope, a mechanical arm, a position sensor and a force sensor, the mechanical arm having a plurality of robot joints thereon, and the neuroendoscope being connected to one end of the mechanical arm, characterized in that, The system performs the following steps: Step S1, determining a target space based on a three-dimensional environment image, determining state space information of a neuroendoscope, and determining action space information of the neuroendoscope; Step S2, determining a reward model of a surgical robot motion action based on a distance between a position of the neuroendoscope and a motion end point in the target space, and determining a penalty model of the surgical robot motion action based on a distance between the neuroendoscope and a key environment structure in the target space and a contact force between the neuroendoscope and a contactable structure in the target space; The step S2 specifically comprises: Step S2-1, determining a to-position reward, wherein the to-position reward is greater when the distance between the neuroendoscope and the motion end point is closer; Step S2-2, determining a collision penalty based on a size relationship between the distance between the neuroendoscope and the key environment structure and a safe distance and a warning distance; Step S2-3, determining a contact force penalty based on a size relationship between the contact force between the neuroendoscope and the contactable structure in the target space and first and second contact force thresholds; Step S3, training, according to the state space information and the action space information and based on the reward model and the penalty model of the surgical robot motion action, using a deep deterministic policy gradient algorithm to obtain an obstacle avoidance control model; The step S3 specifically comprises: Taking the state space of the neuroendoscope and the action space of the neuroendoscope as state space and action space of a DDPG algorithm, taking the reward and the penalty of the surgical robot motion action as reward and penalty of the DDPG algorithm, performing reinforcement learning training, and finally obtaining the obstacle avoidance control model; In the DDPG algorithm reinforcement learning training process, the action acted on the current time state , the next time state was obtained, the current time reward was determined according to the neural endoscope position, key environmental structure position, contact force between the neural endoscope and the contactable structure and the motion end position in the next time state , the current time reward was determined by the current time reward, collision penalty and contact force penalty. Step S4, running a neurosurgical surgical robot, obtaining real-time states from the position sensor and the force sensor, inputting the real-time states into the obstacle avoidance control model, and obtaining control actions; Step S5, controlling the surgical robot by the control actions to complete motion obstacle avoidance of the surgical robot.

2. The neurosurgical procedure robot obstacle avoidance system based on reinforcement learning of claim 1, wherein, The step S1 specifically comprises: Step S1-1, obtaining a tomographic image of an area to be analyzed by a medical imaging device, performing three-dimensional reconstruction on the tomographic image to obtain a three-dimensional model, and determining a target space based on the three-dimensional model, wherein the target space is a motion space of a surgical robot; Step S1-2, determining a position state and a force state of the neuroendoscope as the state space information; Step S1-3, determining a motion state of the mechanical arm as the action space information.

3. The neurosurgical procedure robot obstacle avoidance system based on reinforcement learning of claim 2, wherein, The state space information in step S1-2 specifically comprises: Neuroendoscope position , critical environmental structure position , neuroendoscope contact force with contactable structure , and motion end point position ; The action space information in step S1-3 specifically includes: rotation angle increments of 6 joints of the robot arm 6 .

4. The neurosurgical procedure robot obstacle avoidance system based on reinforcement learning of claim 3, wherein, In step S2-1, the calculation expression of the to-position reward is: wherein, represents a distance to the goal reward, represents a neuroendoscopic position and a motion end position , represents a distance to the goal reward normalization coefficient; In step S2-2, the calculation expression of the collision penalty is: wherein, represents a collision penalty, represents a neuroendoscope position and a critical environmental structure position distance, represents a warning distance, represents a safety distance, represents a collision penalty normalization coefficient; In step S2-3, the calculation expression of the contact force penalty is: wherein, represents a contact force penalty, represents a contact force between the neuroendoscope and the contactable structure, and represent first and second contact force thresholds, respectively, is a contact force penalty normalization coefficient, is a first constant.

5. The reinforcement learning based neurosurgical robot obstacle avoidance system of claim 4, wherein, The specific calculation method for determining a current time reward from a to-position reward, a collision penalty, and a contact force penalty at the current time is: For any time instant, the following expression is used to calculate the reward : wherein, are the on-target reward, collision penalty and contact force penalty, respectively.

6. The reinforcement learning based neurosurgical robot obstacle avoidance system of claim 5, wherein, The step S4 specifically comprises: acquiring the neuroendoscope position in real time through the position sensor , key environmental structure position and motion end position , and acquiring the contact force between the neuroendoscope and the contactable structure in real time through the force sensor as the real-time state; Inputting the real-time state into the obstacle avoidance control model to output corresponding control actions, wherein the control actions include rotation angle increments of each joint of a surgical robot mechanical arm.

Images