Human-robot collaborative intraoperative risk assessment method based on deep inverse reinforcement learning
By constructing a reward and risk assessment neural network through deep inverse reinforcement learning, the problem of real-time assessment of surgical manipulation risks in complex in vivo environments was solved, reducing surgical errors and ensuring surgical safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2022-09-16
- Publication Date
- 2026-07-21
AI Technical Summary
Current technology cannot estimate surgical risks in real time in the complex environment inside the body, resulting in a high risk of misoperation during surgery.
A reward neural network and a risk assessment neural network are constructed using deep inverse reinforcement learning to acquire surgical actions and soft tissue status in real time. The agent network provides virtual path guidance or maintains the current action.
It enables real-time risk assessment in a human-machine collaborative mode, reducing surgical errors and ensuring surgical safety.
Smart Images
Figure CN115312155B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intraoperative risk assessment technology, specifically to a human-machine collaborative intraoperative risk assessment method, system, storage medium, and electronic device based on deep inverse reinforcement learning. Background Technology
[0002] Whether in traditional open surgery or minimally invasive surgery, soft tissue manipulation is a fundamental task within the surgical subtask. Robot-assisted surgery, as an emerging technology, is widely used in various surgical procedures to assist surgeons in performing operations due to its ability to improve surgical precision and dexterity, greatly enhancing surgeons' control over the procedure.
[0003] The constantly changing properties and constraints of soft tissues pose significant risks to robotic surgical operations, especially within the in vivo environment. Existing master-slave operation modes suffer from hand-eye coordination difficulties for surgeons, limited workspaces, restricted stereoscopic visual information, a lack of force feedback and tactile information, and instability and inaccuracy in robotic arm movements caused by manual control. These factors all increase the risk of surgical errors and complicate the procedure. Therefore, it is necessary to improve the autonomous operation capabilities of robots, enabling real-time assessment of operational risks in the complex in vivo environment, thereby assisting surgeons and improving surgical quality. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] To address the shortcomings of existing technologies, this invention provides a human-machine collaborative intraoperative risk assessment method, system, storage medium, and electronic device based on deep inverse reinforcement learning, which solves the technical problem of being unable to estimate the current moment's manipulation risk in complex in vivo environments.
[0006] (II) Technical Solution
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] A human-computer collaborative intraoperative risk assessment method based on deep inverse reinforcement learning is proposed. This method employs deep inverse reinforcement learning to pre-construct a reward neural network and a risk assessment neural network. The method includes:
[0009] S1. Obtain real-time surgical actions, as well as the pre-interaction state consisting of soft tissue state and robot end effector state;
[0010] S2. Based on the real-time surgical actions and pre-interaction state, the reward neural network is used to obtain the real-time reward value;
[0011] S3. Based on the real-time surgical actions, pre-interaction state, and reward value, a risk assessment neural network is used to obtain real-time risk assessment factors.
[0012] S4. Determine whether the real-time risk assessment factor is greater than the set threshold. If so, provide virtual path guidance according to the operation strategy executed by the agent network; otherwise, maintain the current surgical action.
[0013] Preferably, the construction process of the agent network in S4 includes:
[0014] S4a. Construct an expert trajectory pool using an expert strategy;
[0015] S4b. Randomly initialize historical risk hiding information, combine the historical risk hiding information to construct a reward neural network, and construct a risk assessment neural network and an agent network, and initialize the parameters of each network; the agent network includes an action neural network and an evaluation neural network.
[0016] S4c. Under a certain intraoperative task, at the initial moment of the task, construct the trajectory sequence τ of the empty agent; at each moment of the task, obtain the pre-interaction state s composed of the soft tissue state and the robot end effector state. t ;
[0017] S4d, the state before interaction s t The input is fed into the motion neural network, and the output is the velocity vector of the robotic arm's end effector as the motion value a. t ;
[0018] S4e, the action value a t Interact with the environment and acquire a new post-interaction state s, consisting of the organizational state and the state of the robotic arm's end effector. t+1 If state s t+1 If the state is terminated or the maximum task duration limit is reached, then state s will be changed. t+1 As s T Add it to the trajectory sequence τ, making the trajectory sequence τ = {s1, a1, ..., s t a t , ..., s T-1 a T-1 s T}Put it into the agent trajectory pool;
[0019] S4f, Set the state value s t Action value a t and historical risk hidden information h t-1 The input is fed into a reward neural network to obtain the reward value r. t With current risk hidden information h t and hide historical risk information ht-1 Assign the value to the current hidden risk information h t ;
[0020] S4g, the state value s t Action value a t Reward value r t And current risk hidden information h t The input is fed into a risk assessment neural network to obtain the risk assessment factor c. t c t Values in the range of 0 to 1;
[0021] S4h, the state value s t Action value a t Reward value r t Post-interaction state s t+1 Historical risk hidden information h t-1 and risk assessment factor c t Combine them to obtain the experience six-tuple (s t a t r t s t+1 h t-1 c t And put it into the experience pool;
[0022] S4i: When the number of six-tuples in the experience pool reaches the preset capacity, a set number of experience vectors of size batch are randomly extracted from the experience pool to learn the parameters of the action neural network and the evaluation neural network.
[0023] Preferably, the update direction of the agent network in S4 is to maximize the expected reward and minimize the risk assessment value, specifically including:
[0024] The input to the evaluation neural network is the pre-interaction state s. t Action value a t Combined with reward value r t and risk assessment factor c t The output action value evaluates the selected action. The evaluation neural network is updated based on the Bellman equation, and the loss function J... Q Defined as:
[0025]
[0026] The evaluation neural network includes an execution network and a target network; θ represents the parameters of the execution network, and Q... θ (s t a t The value of the action output by the network. For the parameters of the target network, The target network outputs the action value, where γ is the discount factor. This indicates the calculation of the expected value.
[0027] The parameters of the action neural network are optimized based on the action value output by the evaluation neural network, and the loss function J is used. π Defined as:
[0028]
[0029] Where φ represents the parameters of the action neural network.
[0030] Preferably, after updating the parameters of the action neural network and the evaluation neural network, a set number of trajectory experience data of size batch is randomly extracted from the agent's trajectory pool, and the parameters of the reward neural network are obtained using inverse reinforcement learning, specifically including:
[0031] S100. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool.
[0032] S101, Training Reward Function
[0033]
[0034] in, The average reward for each batch of expert trajectories;
[0035]
[0036] The average reward for each agent's trajectory in a batch;
[0037]
[0038] r(s t a t The reward value obtained by the reward neural network for the state-action pair input at each time step is T, where T is the trajectory length and N is the number of trajectories.
[0039] S102. Set the first loss function.
[0040]
[0041] Where w represents the parameters of the reward neural network, and the update formula is based on the loss function, and is: α is the learning rate of the reward neural network.
[0042] Preferably, based on the aforementioned batch of trajectory experience data, the parameters of the risk assessment neural network are obtained by combining inverse reinforcement learning and attention mechanisms, specifically including:
[0043] S200, Input the current state action pair tuple (s t a t The value r is considered the query key Q, used to query the degree of risk that will result from performing the action in the current state; the reward value r is... t Set as the key value K; this reward value incorporates historical risk concealment information h. t-1 Hide current risk information h t Set the V value; and use the Q and K values as inputs to the first multilayer perceptron of the risk assessment neural network, and use the output m of the first multilayer perceptron. t Combined with the V value, it serves as the input to the second multilayer perceptron of the risk assessment neural network;
[0044] S201. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool.
[0045] S202, Training Evaluation Function
[0046]
[0047] in, The average risk assessment value of the expert trajectories for each batch;
[0048]
[0049] For each moment, the risk assessment factor is output by the risk assessment neural network.
[0050] The average risk assessment value of the batch agent trajector ...;
[0051]
[0052] c t For each moment, the risk assessment factor is output by the risk assessment neural network.
[0053] S203, Set the second loss function
[0054]
[0055] v represents the parameters of the risk assessment neural network, and the update formula, based on the loss function, is: β is the learning rate of the risk assessment neural network.
[0056] Preferably, the reward function neural network, risk assessment function neural network, action neural network, or evaluation neural network calculates their respective weight gradients using the gradient chain formula and updates the corresponding network parameters using backpropagation.
[0057] A human-machine collaborative intraoperative risk assessment system based on deep inverse reinforcement learning, employing deep inverse reinforcement learning methods to pre-construct a reward neural network and a risk assessment neural network, the system comprising:
[0058] The acquisition module is used to acquire real-time surgical actions, as well as the pre-interaction state consisting of soft tissue state and robot end effector state;
[0059] The reward module is used to obtain a real-time reward value using the reward neural network based on the real-time surgical actions and the pre-interaction state.
[0060] The evaluation module is used to obtain real-time risk assessment factors using a risk assessment neural network based on the real-time surgical actions, pre-interaction state, and reward value.
[0061] The guidance module is used to determine whether the real-time risk assessment factor is greater than a set threshold. If so, it provides virtual path guidance according to the operation strategy executed by the agent network; otherwise, it maintains the current surgical action.
[0062] A storage medium storing a computer program for human-computer collaborative intraoperative risk assessment based on deep inverse reinforcement learning, wherein the computer program causes a computer to execute the human-computer collaborative intraoperative risk assessment method as described above.
[0063] An electronic device, comprising:
[0064] One or more processors;
[0065] Memory; and
[0066] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing human-machine collaborative intraoperative risk assessment as described above.
[0067] (III) Beneficial Effects
[0068] This invention provides a method, system, storage medium, and electronic device for human-machine collaborative intraoperative risk assessment based on deep inverse reinforcement learning. Compared with existing technologies, it has the following advantages:
[0069] This invention employs a deep inverse reinforcement learning method to pre-construct a reward neural network and a risk assessment neural network. The process includes: acquiring real-time surgical actions and a pre-interaction state composed of soft tissue state and robot end effector state; obtaining a real-time reward value using the reward neural network based on the real-time surgical actions and pre-interaction state; obtaining a real-time risk assessment factor using the risk assessment neural network based on the real-time surgical actions, pre-interaction state, and reward value; determining whether the real-time risk assessment factor is greater than a set threshold; if so, providing virtual path guidance based on the agent network's execution strategy; otherwise, maintaining the current surgical action. In human-machine collaboration mode, it can obtain real-time risk estimates for actions taken in the current state and provide real-time guidance, reducing surgical errors, minimizing intraoperative risks, and ensuring surgical safety. Attached Figure Description
[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0071] Figure 1 A flowchart illustrating a human-machine collaborative intraoperative risk assessment method based on deep inverse reinforcement learning, provided in an embodiment of the present invention;
[0072] Figure 2 The present invention provides an overall framework diagram of a human-machine collaborative intraoperative risk assessment method based on deep inverse reinforcement learning. Detailed Implementation
[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0074] This application provides a human-machine collaborative intraoperative risk assessment method, system, storage medium, and electronic device based on deep inverse reinforcement learning, which solves the technical problem of being unable to estimate the current moment's manipulation risk in complex in vivo environments.
[0075] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:
[0076] Existing reinforcement learning-based methods for performing intraoperative tasks, employing discrete actions, cannot effectively handle manipulation in high-dimensional environments. Furthermore, training execution strategies using hand-designed reward functions is insufficient to encompass all situations in the complex surgical environment, limiting the algorithm's practical application. Additionally, current surgical risk assessment technologies primarily focus on pre- or post-operative risk control, or on evaluating intraoperative techniques (simply categorized by corresponding skill levels). However, robotic autonomous surgery requires real-time estimation of manipulation risks within the complex intraoperative environment, yet existing technologies lack specific techniques for risk assessment during human-machine-assisted surgical procedures.
[0077] Based on this, the present invention proposes a human-machine collaborative intraoperative risk assessment scheme applicable to, but not limited to, soft tissue manipulation scenarios. Specifically, it includes: acquiring real-time surgical actions, and a pre-interaction state composed of the soft tissue state and the robot end effector state; obtaining a real-time reward value using the reward neural network based on the real-time surgical actions and the pre-interaction state; obtaining a real-time risk assessment factor using a risk assessment neural network based on the real-time surgical actions, the pre-interaction state, and the reward value; determining whether the real-time risk assessment factor is greater than a set threshold; if so, providing virtual path guidance according to the agent network's execution operation strategy; otherwise, maintaining the current surgical actions.
[0078] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0079] Example:
[0080] like Figure 1 As shown, this invention provides a human-machine collaborative intraoperative risk assessment method based on deep inverse reinforcement learning. The method pre-constructs a reward neural network and a risk assessment neural network using deep inverse reinforcement learning. The method includes:
[0081] S1. Obtain real-time surgical actions, as well as the pre-interaction state consisting of soft tissue state and robot end effector state;
[0082] S2. Based on the real-time surgical actions and pre-interaction state, the reward neural network is used to obtain the real-time reward value;
[0083] S3. Based on the real-time surgical actions, pre-interaction state, and reward value, a risk assessment neural network is used to obtain real-time risk assessment factors.
[0084] S4. Determine whether the real-time risk assessment factor is greater than the set threshold. If so, provide virtual path guidance according to the operation strategy executed by the agent network; otherwise, maintain the current surgical action.
[0085] In the human-machine collaboration mode, the embodiments of the present invention can obtain the risk estimate of the current state and take action in real time, and provide real-time guidance to reduce the doctor's operation error during the operation, reduce the risk of operation during the operation, and ensure the safety of the operation.
[0086] The above technical solution will be explained in detail below:
[0087] First, it should be noted that before applying the aforementioned risk assessment method for human-machine collaboration, it is necessary to pre-construct the intelligent agent network mentioned in S4, specifically including:
[0088] S4a. Construct an expert trajectory pool using an expert strategy;
[0089] Using sensors to record the surgeon's hand movements and tissue conditions during a specific intraoperative task, an expert trajectory consisting of a series of state-action pairs is obtained. Stored in the expert trajectory pool. This represents the state at time t in the i-th trajectory. Let T represent the action at time t in the i-th trajectory, T represent the length of the trajectory, and N represent the number of trajectories. There are N expert trajectories. As expert knowledge for inverse reinforcement learning.
[0090] S4b. Randomly initialize historical risk hiding information, combine the historical risk hiding information to construct a reward neural network, and construct a risk assessment neural network and an agent network, and initialize the parameters of each network; the agent network includes an action neural network and an evaluation neural network.
[0091] S4c. Under a certain intraoperative task, at the initial moment of the task, construct the trajectory sequence τ of the empty agent; at each moment of the task, obtain the pre-interaction state s composed of the soft tissue state and the robot end effector state. t ;
[0092] S4d, the state before interaction s t The input is fed into the motion neural network, and the output is the velocity vector of the robotic arm's end effector as the motion value a. t ;
[0093] S4e, the action value a t Interact with the environment and acquire a new post-interaction state s, consisting of the organizational state and the state of the robotic arm's end effector. t+1 If state s t+1If the state is terminated or the maximum task duration limit is reached, then state s will be changed. t+1 As s T Add it to the trajectory sequence τ, making the trajectory sequence τ = {s1, a1, ..., s t a t , ..., s T-1 a T-1 s T}Put it into the agent trajectory pool;
[0094] S4f, Set the state value s t Action value a t and historical risk hidden information h t-1 The input is fed into a reward neural network to obtain the reward value r. t With current risk hidden information h t and hide historical risk information h t-1 Assign the value to the current hidden risk information h t ;
[0095] S4g, the state value s t Action value a t Reward value r t And current risk hidden information h t The input is fed into a risk assessment neural network to obtain the risk assessment factor c. t c t Values in the range of 0 to 1;
[0096] S4h, the state value s t Action value a t Reward value r t Post-interaction state s t+1 Historical risk hidden information h t-1 and risk assessment factor c t Combine them to obtain the experience six-tuple (s t a t r t s t+1 h t-1 c t And put it into the experience pool;
[0097] S4i: When the number of six-tuples in the experience pool reaches the preset capacity, a set number of experience vectors of size batch are randomly extracted from the experience pool to learn the parameters of the action neural network and the evaluation neural network.
[0098] Furthermore, the update direction of the agent network in S4 is to maximize the expected reward and minimize the risk assessment value, specifically including:
[0099] The input to the evaluation neural network is the pre-interaction state s. t Action value a t Combined with reward value r t and risk assessment factor c t The output action value evaluates the selected action. The evaluation neural network is updated based on the Bellman equation, and the loss function J... Q Defined as:
[0100]
[0101] The evaluation neural network includes an execution network and a target network; θ represents the parameters of the execution network, and Q... θ (s t a t The value of the action output by the network. For the parameters of the target network, The target network outputs the action value, where γ is the discount factor. This indicates the calculation of the expected value.
[0102] The parameters of the action neural network are optimized based on the action value output by the evaluation neural network, and the loss function J is used. π Defined as:
[0103]
[0104] Where φ represents the parameters of the action neural network.
[0105] Secondly, after updating the parameters of the action neural network and the evaluation neural network, a set number of trajectory experience data of size batch is randomly extracted from the agent's trajectory pool. Inverse reinforcement learning is then used to obtain the parameters of the reward neural network, specifically including:
[0106] S100. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool.
[0107] S101, Training reward function (It is hoped that this reward function will give more feedback value to expert trajectory data and less reward value to trajectory data generated by the agent);
[0108]
[0109] in, The average reward for each batch of expert trajectories;
[0110]
[0111] The average reward for each agent's trajectory in a batch;
[0112]
[0113] r(s t a t The reward value obtained by the reward neural network for the state-action pair input at each time step is T, where T is the trajectory length and N is the number of trajectories.
[0114] Therefore, the embodiments of the present invention no longer use a manually designed reward function, but instead learn the interaction relationship with the environment autonomously based on the network.
[0115] S102. Set the first loss function (hoping that the reward obtained by the trained agent on the trajectory is close to the reward obtained on the expert trajectory, so that the agent can learn the decision-making behavior of the expert).
[0116]
[0117] Where w represents the parameters of the reward neural network, and the update formula is based on the loss function, and is: α is the learning rate of the reward neural network.
[0118] Finally, based on the aforementioned batch of trajectory experience data, the parameters of the risk assessment neural network are obtained by combining inverse reinforcement learning and attention mechanisms, specifically including:
[0119] S200, such as Figure 2 As shown, the input current state action pair tuple (s) t a t The value r is considered the query key Q, used to query the degree of risk that will result from performing the action in the current state; the reward value r is... t Set as the key value K; this reward value incorporates historical risk concealment information h. t-1 Hide current risk information h t Set the V value; and use the Q and K values as inputs to the first multilayer perceptron of the risk assessment neural network, and use the output m of the first multilayer perceptron. t Combined with the V value, it serves as the input to the second multilayer perceptron of the risk assessment neural network;
[0120] S201. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool.
[0121] S202, Training the evaluation function (hoping that the risk assessment network will have a lower risk assessment value for expert trajectory data and a higher risk assessment value for trajectory data generated by the agent);
[0122]
[0123] in, The average risk assessment value of the expert trajectories for each batch;
[0124]
[0125] For each moment, the risk assessment factor is output by the risk assessment neural network.
[0126] The average risk assessment value of the batch agent trajector ...;
[0127]
[0128] c t For each moment, the risk assessment factor is output by the risk assessment neural network.
[0129] S203. Set a second loss function (hoping that the risk assessment value of the trajectory obtained by the trained agent is close to the risk assessment value obtained by the expert trajectory, so that the agent learns the risk assessment behavior of the expert).
[0130]
[0131] v represents the parameters of the risk assessment neural network, and the update formula, based on the loss function, is: β is the learning rate of the risk assessment neural network.
[0132] For reference, the reward function neural network, risk assessment function neural network, action neural network, or evaluation neural network mentioned above can all calculate their respective weight gradients using the gradient chain formula, and update the corresponding network parameters using backpropagation until the set maximum number of steps or the set convergence target is reached. At this point, training ends, and the parameters of each trained neural network are saved.
[0133] In step S1, real-time surgical actions are acquired, along with the pre-interaction state consisting of the soft tissue state and the robot end effector state.
[0134] Similar to the training phase, this step calculates a dense 3D point cloud from the 3D soft tissue images obtained by binocular endoscopy, and performs dimensionality reduction processing on the point cloud to obtain the soft tissue state. The robot end effector state is obtained by reading the robot end effector parameters.
[0135] In step S2, the reward neural network is used to obtain the real-time reward value based on the real-time surgical actions and the pre-interaction state.
[0136] Specifically, such as Figure 2As shown, the reward neural network can be composed of multilayer perceptrons and gated recurrent units stacked in the order of multilayer perceptron, gated recurrent units, and multilayer perceptrons.
[0137] The multilayer perceptron consists of two stacked 256-dimensional fully connected layers, with a ReLU activation function applied after each fully connected layer. The input to the gated recurrent unit is the output of the previous multilayer perceptron and the historical risk hiding information h. t-1 The output of the gated recurrent unit is divided into two parts: one part is the risk hiding information ht of the current state execution action, and the other part is the input of the next layer multilayer perceptron combined with the past risk hiding information; the second multilayer perceptron is composed of two 256-dimensional fully connected layers stacked together. The first fully connected layer uses the ReLU activation function, and the output of the second fully connected layer is the reward value rt of the (batch, 1)-dimensional structure.
[0138] Therefore, it can be seen that the embodiments of the present invention redesign the reward based on the relationship between risk and reward, and the reward takes into account risk factors, making the acquisition of the reward more cautious.
[0139] In step S3, a risk assessment neural network is used to obtain real-time risk assessment factors based on the real-time surgical actions, pre-interaction state, and reward value.
[0140] Specifically, such as Figure 2 As shown, the risk assessment function neural network consists of two multilayer perceptrons. The first multilayer perceptron is composed of two stacked 256-dimensional fully connected layers, with a ReLU activation function following each fully connected layer. The second multilayer perceptron is composed of two stacked 256-dimensional fully connected layers, with a ReLU activation function following the first fully connected layer, and a sigmoid layer following the second fully connected layer to normalize the output. The output is a risk assessment factor c with a (batch, 1)-dimensional structure. t c t Values in the range of 0 to 1.
[0141] In step S4, it is determined whether the real-time risk assessment factor is greater than the set threshold. If so, virtual path guidance is given according to the operation strategy executed by the intelligent agent network (the doctor can choose to refer to or not refer to this opinion and make the current decision and generate the action); otherwise, the current surgical action is maintained.
[0142] During the training phase of the agent network, it is clearly defined that it includes an action neural network and an evaluation neural network. To further clarify:
[0143] The action neural network includes fully connected layers and tanh layers, which are stacked in the order of the fully connected layers. Each fully connected layer is followed by a ReLU activation function, and the last fully connected layer is followed by a tanh layer.
[0144] The evaluation neural network includes an execution network and a target network, which have the same structure. The execution network includes fully connected layers stacked in order, with each fully connected layer followed by a ReLU activation function. Specifically, the target network includes fully connected layers stacked in order, with each fully connected layer followed by a ReLU activation function. The execution network and the target network have the same network structure.
[0145] As mentioned above, the action neural network and evaluation neural network can be embedded using general reinforcement learning algorithms as a general module, and will not be elaborated further here.
[0146] This invention provides a human-machine collaborative intraoperative risk assessment system based on deep inverse reinforcement learning. The system employs deep inverse reinforcement learning to pre-construct a reward neural network and a risk assessment neural network. The system includes:
[0147] The acquisition module is used to acquire real-time surgical actions, as well as the pre-interaction state consisting of soft tissue state and robot end effector state;
[0148] The reward module is used to obtain a real-time reward value using the reward neural network based on the real-time surgical actions and the pre-interaction state.
[0149] The evaluation module is used to obtain real-time risk assessment factors using a risk assessment neural network based on the real-time surgical actions, pre-interaction state, and reward value.
[0150] The guidance module is used to determine whether the real-time risk assessment factor is greater than a set threshold. If so, it provides virtual path guidance according to the operation strategy executed by the agent network; otherwise, it maintains the current surgical action.
[0151] This invention provides a storage medium storing a computer program for human-computer collaborative intraoperative risk assessment based on deep inverse reinforcement learning, wherein the computer program causes a computer to execute the human-computer collaborative intraoperative risk assessment method described above.
[0152] This invention provides an electronic device, comprising:
[0153] One or more processors;
[0154] Memory; and
[0155] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing human-machine collaborative intraoperative risk assessment as described above.
[0156] It is understood that the human-machine collaborative intraoperative risk assessment system, storage medium and electronic device based on deep inverse reinforcement learning provided in the embodiments of the present invention correspond to the human-machine collaborative intraoperative risk assessment method provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the human-machine collaborative intraoperative risk assessment method, and will not be repeated here.
[0157] In summary, compared with existing technologies, it has the following beneficial effects:
[0158] 1. In the human-machine collaboration mode, the embodiments of the present invention can obtain the risk estimate of the current state and take action in real time, and can provide guidance in real time, reduce the doctor's operation error during the operation, reduce the risk of operation during the operation, and ensure the safety of the operation.
[0159] 2. In this embodiment of the invention, the reward function is no longer designed manually, but the network learns the interaction relationship with the environment autonomously.
[0160] 3. The embodiments of the present invention redesign the reward based on the relationship between risk and reward, and the reward takes into account risk factors, making the acquisition of reward more cautious.
[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0162] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A human-machine collaborative intraoperative risk assessment system based on deep inverse reinforcement learning, characterized in that, The system employs a deep inverse reinforcement learning approach to pre-construct reward neural networks and risk assessment neural networks. The system includes: The acquisition module is used to acquire real-time surgical actions, as well as the pre-interaction state consisting of soft tissue state and robot end effector state; The reward module is used to obtain a real-time reward value using the reward neural network based on the real-time surgical actions and the pre-interaction state. The evaluation module is used to obtain real-time risk assessment factors using a risk assessment neural network based on the real-time surgical actions, pre-interaction state, and reward value. The guidance module is used to determine whether the real-time risk assessment factor is greater than a set threshold. If so, it provides virtual path guidance according to the operation strategy executed by the intelligent agent network; otherwise, it maintains the current surgical action. The construction process of the intelligent agent network includes: S4a. Construct an expert trajectory pool using an expert strategy; S4b. Randomly initialize historical risk hiding information, combine the historical risk hiding information to construct a reward neural network, and construct a risk assessment neural network and an agent network, and initialize the parameters of each network; the agent network includes an action neural network and an evaluation neural network. S4c, Under a certain intraoperative task, at the initial moment of the task, construct the trajectory sequence of the empty agent. At each moment of the task, acquire the pre-interaction state, which consists of the soft tissue state and the robot's end effector state. ; S4d, the state before interaction The input is fed into the motion neural network, and the output is the velocity vector of the robotic arm's end effector as the motion value. ; S4e, the action value Interact with the environment and acquire a new post-interaction state consisting of the organizational state and the state of the robotic arm's end effector. If the state If the state is terminated or the maximum task duration limit is reached, then the state will be changed. As Add to trajectory sequence In the middle, the trajectory sequence Add to the agent's trajectory pool; S4f, the state value Action value and historical risk hidden information The input is fed into the reward neural network to obtain the reward value. Information hidden with current risks and hide historical risk information. Assigning values to current hidden risk information ; S4g, the state value Action value Reward Value and current risk hidden information The input is fed into a risk assessment neural network to obtain risk assessment factors. , Values in the range of 0 to 1; S4h, the state value Action value Reward Value Post-interaction state Historical risk hidden information and risk assessment factors Combine them to obtain an experience six-tuple. And put it into the experience pool; S4i: When the number of six-tuples in the experience pool reaches the preset capacity, a set number of experience vectors of size batch are randomly extracted from the experience pool to learn the parameters of the action neural network and the evaluation neural network.
2. The human-machine collaborative intraoperative risk assessment system as described in claim 1, characterized in that, The update direction of the agent network is to maximize the expected reward and minimize the risk assessment value, specifically including: The input to the evaluation neural network is the state before the interaction. Action value Combined with reward value and risk assessment factors The output action value evaluates the selected action. The evaluation neural network is updated based on the Bellman equation, and the loss function is... Defined as: The evaluation neural network includes the execution network and the target network; To execute the network parameters, To execute the action value output by the network, For the parameters of the target network, The action value output by the target network. As a discount factor, This indicates the calculation of the expected value. The parameters of the action neural network are optimized based on the action value output by the evaluation neural network, and the loss function is used. Defined as: in, These are the parameters of the action neural network.
3. The human-machine collaborative intraoperative risk assessment system as described in claim 2, characterized in that, After updating the parameters of the action neural network and the evaluation neural network, a set number of trajectory experience data of size batch is randomly extracted from the agent's trajectory pool. Inverse reinforcement learning is then used to obtain the parameters of the reward neural network, specifically including: S100. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool. S101, Training Reward Function in, The average reward for each batch of expert trajectories; The average reward for each agent's trajectory in a batch; The reward neural network receives the reward value for the state-action pair input at each time step. T For the trajectory length, N The number of trajectories; S102. Set the first loss function. in, To reward the neural network parameters, the update formula is based on the loss function, and is as follows: , The learning rate is used to reward the neural network.
4. The human-machine collaborative intraoperative risk assessment system as described in claim 3, characterized in that, Based on the aforementioned batch-sized trajectory experience data, the parameters of the risk assessment neural network are obtained by combining inverse reinforcement learning and attention mechanisms, specifically including: S200, Input the current state action pair tuple Treat it as the query key Q, used to query the degree of risk that will result from performing this action in the current state; and the reward value... Set to a key value K; this reward value incorporates historical risk concealment information. Hide current risk information Set the V value; and use the Q and K values as inputs to the first multilayer perceptron of the risk assessment neural network, and use the output of the first multilayer perceptron... Combined with the V value, it serves as the input to the second multilayer perceptron of the risk assessment neural network; S201. Randomly extract trajectory experience data of batch size from the expert trajectory pool and the agent trajectory pool. S202, Training Evaluation Function in, The average risk assessment value of the expert trajectories for each batch; For each moment, the risk assessment factor is output by the risk assessment neural network. The average risk assessment value of the batch agent trajector ...; For each moment, the risk assessment factor is output by the risk assessment neural network. S203, Set the second loss function The parameters of the risk assessment neural network are updated using a formula based on the loss function, as follows: , The learning rate of the risk assessment neural network.
5. The human-machine collaborative intraoperative risk assessment system as described in any one of claims 1 to 4, characterized in that, The reward function neural network, risk assessment function neural network, action neural network, or evaluation neural network calculates their respective weight gradients using the gradient chain formula and updates the corresponding network parameters using backpropagation.
6. A storage medium, characterized in that, It stores a computer program for human-machine collaborative intraoperative risk assessment based on deep inverse reinforcement learning, wherein the computer program enables the computer to perform all the module functions of the human-machine collaborative intraoperative risk assessment system as described in any one of claims 1 to 5.
7. An electronic device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including all module functions for performing the human-machine collaborative intraoperative risk assessment system as described in any one of claims 1 to 5.