Control devices and methods for vascular interventional surgery

Abstract

This invention provides a control device and method for vascular interventional surgery, relating to the field of control technology. The device includes: an acquisition module for acquiring first image data from the previous moment, second image data from the current moment, and first action information from the previous moment; both the first and second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery; a prediction module for obtaining the probability of selecting each action at the current moment, output by a target surgical operation model; the target surgical operation model is trained using an offline reinforcement learning method; a determination module for determining a target control command based on the probability of selecting each action; and a control module for controlling the movement of the instrument within the blood vessel based on the target control command. This invention improves the accuracy of the trained target surgical operation model and further improves the accuracy of instrument movement within the blood vessel.

Description

Technical Field

[0001] This invention relates to the field of control technology, and in particular to a control device and method for vascular interventional surgery. Background Technology

[0002] Interventional vascular surgery is a minimally invasive treatment method that utilizes a robotic system. Guided by an imaging system, the surgeon operates the robotic system to control interventional instruments such as guide wires to reach the lesion site through the blood vessel cavity, performing treatments such as dissolving thrombi and dilating narrowed blood vessels.

[0003] In related technologies, vascular interventional surgery operation models are typically trained using imitation learning or statistical learning methods based on doctors' operation examples, thereby enabling autonomous instrument delivery by vascular interventional surgery robots.

[0004] However, in the aforementioned related technologies, the model is trained using imitation learning or statistical learning methods. The optimization of the model parameters is aimed at the doctor's operation examples. Therefore, the accuracy of the model depends on the quality of the doctor's operation examples. If the quality of the doctor's operation examples is poor, it will lead to a decrease in the accuracy of the trained model, which in turn reduces the accuracy of the instrument's movement in the blood vessel. Summary of the Invention

[0005] To address the problems existing in the prior art, embodiments of the present invention provide a control device and method for vascular interventional surgery.

[0006] This invention provides a control device for vascular interventional surgery, comprising:

[0007] The acquisition module is used to acquire the first image data of the previous moment, the second image data of the current moment, and the first action information of the previous moment; both the first image data and the second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery;

[0008] The prediction module is used to input the first image data, the second image data, and the first action information into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained based on an offline reinforcement learning method using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data; the first image sample data is the data collected at the moment preceding the second image sample data.

[0009] A determination module is used to determine a target control command based on the probability of selecting each of the actions;

[0010] A control module is used to control the movement of the device in the blood vessel based on the target control command.

[0011] According to the present invention, a control device for vascular interventional surgery is provided, wherein the target surgical operation model is trained based on the following method:

[0012] The first image sample data and the second image sample data in the sample data are input into the encoder of the initial surgical operation model to obtain the encoded information sample output by the encoder;

[0013] The encoded information samples and the action sample information in the sample data are input into the strategy estimation sub-model and function estimation sub-model of the initial surgical operation model to obtain the predicted probability of selecting each action at the first moment output by the strategy estimation sub-model and the function estimate value output by the function estimation sub-model; the first moment is the moment corresponding to the acquisition of the second image sample data;

[0014] The target action is determined based on the predicted probability of each action selected at the first moment, and the reward value at the first moment is determined based on the target action.

[0015] The model parameters of the initial surgical operation model are updated based on the reward value and the function estimate to obtain the target surgical operation model; the target surgical operation model corresponds to the maximum cumulative reward value.

[0016] According to a control device for vascular interventional surgery provided by the present invention, the step of updating the model parameters of the initial surgical operation model based on the reward value and the function estimate to obtain the target surgical operation model includes:

[0017] Based on the reward value, the estimated value of the function, the estimated value of the objective function, and the advantage of the sample data relative to the agent's policy, the function estimation loss function is determined;

[0018] The policy mimicry loss function is determined based on the agent's policy and the sample data.

[0019] Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain the target surgical operation model.

[0020] According to a control device for vascular interventional surgery provided by the present invention, the step of optimizing the model parameters of the initial surgical operation model based on the function estimation loss function and the strategy imitation loss function to obtain the target surgical operation model includes:

[0021] Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain an imitation-based surgical operation model;

[0022] Based on the agent's policy and the estimated value of the objective function, the policy optimization loss function is determined;

[0023] Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

[0024] According to a control device for vascular interventional surgery provided by the present invention, the step of determining the reward value at the first moment based on the target action includes:

[0025] Perform the target action to obtain the position of the device in the blood vessel at a first moment;

[0026] The reward value at the first moment is determined based on whether the position deviates from the target path, whether the contact force of the device is greater than or equal to a preset threshold, and the difference between the position at the first moment and the position at the second moment when the device moves from the position at the first moment to the position at the second moment.

[0027] According to a control device for vascular interventional surgery provided by the present invention, the step of optimizing the model parameters of the imitation-based surgical operation model based on the strategy optimization loss function and the function estimation loss function to obtain the target surgical operation model includes:

[0028] The sampling probability of the corresponding sample data is determined based on the weight of each sample data.

[0029] The target sample data is determined based on the sampling probability of each of the sample data.

[0030] The target sample data is input into the imitation-based surgical operation model. Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

[0031] According to the present invention, a control device for vascular interventional surgery further includes:

[0032] The update module is used to update the weights of the target sample data based on the reward value, the estimated value of the objective function, and the estimated value of the function after optimizing the model parameters of the imitation-based surgical operation model in each round.

[0033] The present invention also provides a control method for vascular interventional surgery, comprising: acquiring first image data at the previous moment, second image data at the current moment, and first action information at the previous moment; the first image data and the second image data both include vascular image data and instrument image data corresponding to the vascular interventional surgery;

[0034] The first image data, the second image data, and the first action information are input into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data, based on an offline reinforcement learning method; the first image sample data is the data collected at the moment preceding the second image sample data.

[0035] The target control command is determined based on the probability of selecting each of the actions;

[0036] The device is controlled to move within the blood vessel based on the target control command.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method for vascular interventional surgery as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method for vascular interventional surgery as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the control method for vascular interventional surgery as described above.

[0040] The control device and method for vascular interventional surgery provided by this invention input the first image data from the previous moment, the second image data from the current moment, and the first action information from the previous moment into a target surgical operation model. This target surgical operation model can output the probability of instrument actions being selected, thereby determining the target control command corresponding to the instrument's action, and then controlling the movement of the instrument in the blood vessel according to the target control command. Since this target surgical operation model is trained using an offline reinforcement learning method, which optimizes model parameters based on a reward mechanism and does not depend on the quality of the physician's operation examples, the accuracy of the trained target surgical operation model is improved, further enhancing the accuracy of instrument movement in the blood vessel. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the control device for vascular interventional surgery provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the data acquisition device provided by the present invention;

[0044] Figure 3 This is a schematic diagram of the surgical operation model provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the encoder structure in the surgical operation model provided by the present invention;

[0046] Figure 5 This is a schematic diagram illustrating the interaction between the surgical operation model and the surgical environment provided by the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the process of optimizing model parameters for a surgical operation model based on imitation, as provided by the present invention.

[0048] Figure 7 This is a flowchart illustrating the control method for vascular interventional surgery provided by the present invention;

[0049] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined Figures 1-6 The present invention describes a control device for vascular interventional surgery.

[0052] Figure 1 This is a schematic diagram of the structure of the control device for vascular interventional surgery provided in an embodiment of the present invention, as shown below. Figure 1As shown, the control device 100 for the vascular interventional surgery includes an acquisition module 101, a prediction module 102, a determination module 103, and a control module 104; wherein:

[0053] The acquisition module 101 is used to acquire the first image data of the previous moment, the second image data of the current moment, and the first action information of the previous moment; the first image data and the second image data both include vascular image data and instrument image data corresponding to the vascular interventional surgery.

[0054] For example, the following is combined with Figure 2 The diagram illustrates the structure of the data acquisition device, and explains the first and second image data.

[0055] Figure 2 The operational example acquisition device includes a master end 21 of a vascular interventional surgical robot, a slave end 22 of a vascular interventional surgical robot, a guidewire 23, a catheter 24, a vascular model 25, a camera 26, a control handle 27, and a display device 28. The vascular model 25 can be manufactured using 3D printing, and the camera 26 is located above the vascular model 25 and its position is fixed. The slave end 22 of the vascular interventional surgical robot can receive control commands from the master end 21 of the vascular interventional surgical robot and manipulate the guidewire 23 to move within the vascular model 25 according to the control commands. In the following embodiments, the master end 21 of the vascular interventional surgical robot is referred to as the master end, and the slave end 22 of the vascular interventional surgical robot is referred to as the slave end.

[0056] Camera 26 is used to capture images of blood vessels and instruments. The main device can acquire the captured images from camera 26, perform binarization processing on the captured images, and send the binarized images to display device 28. The first image data and the second image data in this invention both include binarized blood vessel image data and instrument image data, i.e., binarized image data of blood vessels and binarized image data of instruments. The instruments may include guidewires; this invention does not specifically limit the type of instrument.

[0057] The binarized image data of blood vessels can be obtained as follows: In the 3D printed blood vessel model 25, the colors of the blood vessel portion and the background portion are set to be different, resulting in different pixel values ​​for the blood vessel portion and the background portion. The master terminal can set a pixel threshold range for the blood vessel portion and determine whether the captured image falls within this range. If it does, the pixels within the threshold range can be extracted and used as the binarized image data of the blood vessel. The binarized image data of the instrument is obtained based on the difference between an image including both the instrument and blood vessels and an image containing only blood vessels. Furthermore, after obtaining the binarized image data of the instrument, the master terminal can perform a closing operation on the binarized image data of the instrument to eliminate any possible breaks in the instrument in the binarized image. Additionally, the master terminal can perform centering processing on the binarized image data of the instrument, making the resulting model applicable to instruments of different thicknesses.

[0058] Furthermore, the first motion information can be the actions performed by the vascular interventional surgical robot in the blood vessel based on control commands. The control commands include control commands for two degrees of freedom: axial and rotational. Axial freedom control commands can include constant-speed forward and constant-speed backward commands. Rotational freedom control commands can include no-rotation, clockwise rotation, and counter-clockwise rotation commands. The speeds corresponding to the clockwise and counter-clockwise rotation commands can be set according to requirements. Specifically, a constant-speed forward command corresponds to the instrument moving forward at a constant speed in the blood vessel, a constant-speed backward command corresponds to the instrument moving backward at a constant speed in the blood vessel, a no-rotation command corresponds to the instrument not rotating in the blood vessel, a clockwise rotation command corresponds to the instrument rotating clockwise in the blood vessel at a set speed, and a counter-clockwise rotation command corresponds to the instrument rotating counter-clockwise in the blood vessel at a set speed. The duration of the control commands can be set according to actual needs; in this invention, the duration of the control commands can be 0.5 seconds.

[0059] The prediction module 102 is used to input the first image data, the second image data, and the first action information into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained based on an offline reinforcement learning method using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data; the first image sample data is the data collected at the moment preceding the second image sample data.

[0060] For example, the first image data, the second image data, and the first motion information are input into the target surgical operation model. The target surgical operation model can output the probability of selecting each motion at the current moment. For instance, the target surgical operation model can output a 97% probability of the instrument moving forward, a 10% probability of the instrument moving backward, a 70% probability of the instrument rotating clockwise, and a 5% probability of the instrument rotating counterclockwise.

[0061] In the process of training the target surgical operation model using offline reinforcement learning methods and utilizing first image sample data, second image sample data, and the corresponding action sample information of the first image sample data, the first image sample data can refer to the description of first image data in the above embodiments, the second image sample data can refer to the description of second image data in the above embodiments, and the corresponding action sample information of the first image sample data can refer to the first action information in the acquisition module of the above embodiments. The first image sample data differs from the first image data in that it is the model input data during training the surgical operation model, while the second image data is the model input data during using the surgical operation model. Similarly, the second image sample data differs from the second image data in that it is the model input data during training the surgical operation model, while the second image data is the model input data during using the surgical operation model.

[0062] It should be noted that the target surgical operation model was trained using an offline reinforcement learning method. Offline reinforcement learning is a machine learning method that improves the agent's policy based on given data to maximize the expected cumulative reward obtained by the agent.

[0063] The determination module 103 is used to determine the target control command based on the probability of selecting each of the actions.

[0064] For example, after the target surgical operation model outputs the probability of the selected action, the control command corresponding to the action with the highest probability can be determined as the target control command. For instance, if the action with the highest probability is instrument advancement, then the target control command corresponding to the advancement action is the advancement command.

[0065] The control module 104 is used to control the movement of the device in the blood vessel based on the target control command.

[0066] For example, after the target control command corresponding to the selected action is determined in the determination module, the vascular interventional surgical robot can control the movement of instruments in the blood vessel based on the target control command.

[0067] The control device for vascular interventional surgery provided by this invention inputs the first image data from the previous moment, the second image data from the current moment, and the first action information from the previous moment into a target surgical operation model. This target surgical operation model can output the probability of instrument actions being selected, thereby determining the target control command corresponding to the instrument's action, and then controlling the movement of the instrument in the blood vessel according to the target control command. Since this target surgical operation model is trained using an offline reinforcement learning method, which optimizes model parameters based on a reward mechanism and does not depend on the quality of the physician's operation examples, the accuracy of the trained target surgical operation model is improved, further enhancing the accuracy of instrument movement in the blood vessel.

[0068] In one embodiment, the target surgical operation model in the prediction module 102 is trained in the following manner:

[0069] The first image sample data and the second image sample data in the sample data are input into the encoder of the initial surgical operation model to obtain the encoded information sample output by the encoder;

[0070] The encoded information samples and the action sample information in the sample data are input into the strategy estimation sub-model and function estimation sub-model of the initial surgical operation model to obtain the predicted probability of selecting each action at the first moment output by the strategy estimation sub-model and the function estimate value output by the function estimation sub-model; the first moment is the moment corresponding to the acquisition of the second image sample data;

[0071] The target action is determined based on the predicted probability of each action selected at the first moment, and the reward value at the first moment is determined based on the target action.

[0072] The model parameters of the initial surgical operation model are updated based on the reward value and the function estimate to obtain the target surgical operation model; the target surgical operation model corresponds to the maximum cumulative reward value.

[0073] For example, the following is combined with Figure 3 The structural diagram of the surgical operation model illustrates the training process of the surgical operation model of the present invention.

[0074] Figure 3 The surgical operation model includes encoder 301, policy estimation sub-model 302, and function estimation sub-model 303.

[0075] The encoder 301 is used to receive the first image sample data and the second image sample data in the sample data, and to compress the first image sample data and the second image sample data to obtain the encoded sample information.

[0076] The policy estimation sub-model 302 receives encoded sample information and action sample information from the sample data, and predicts the probability of selecting each action at the first moment based on the encoded sample information, action sample information, and the set agent policy. The target action can be determined based on the predicted probability of selecting each action at the first moment, and the reward value at the first moment can be determined based on the target action. The device action can be determined based on the predicted probability of each action; the description of determining the device action in the above embodiments is as described and will not be repeated here.

[0077] The function estimation sub-model 303 receives encoded sample information and action sample information from the sample data, and calculates function estimates based on the encoded sample information and action sample information. Optionally, the function estimation sub-model may include two local Q-function estimators. The two local Q-function estimators have the same model structure, but different parameters are set within them, resulting in two different estimates of the Q-function output by the two local Q-function estimators. The two estimates can be expressed as follows: and The estimated value of the function is given by and The minimum value of is obtained, and the function estimate satisfies the following formula (1):

[0078] (1);

[0079] The model parameters of the initial surgical procedure model can be updated based on the reward value and function estimate, thereby obtaining the target surgical procedure model.

[0080] The structure of the encoder in the surgical operation model of the present invention is as follows: Figure 4 As shown, the encoder includes: convolutional layers, nonlinear layers, and a regularization layer (Adaptive Local Signal Mixing, A-LIX). The convolutional and nonlinear layers can be considered as one layer. The A-LIX layer performs a random translation-like operation on the feature map obtained from the convolutional layers to smooth the gradient of the convolutional layers.

[0081] During the training of the surgical operation model, the A-LIX layer in the encoder converts the feature maps output by the convolutional layer. As the input to the A-LIX layer, the input to the A-LIX layer can be denoted as: ,in, Indicates the number of channels in the convolutional layer. Indicates the height of the feature map, This represents the width of the feature map. The A-LIX layer starts from a uniform distribution... Mid-sampling and ,in, Indicates the translation range. This represents the offset in the height direction. This indicates the offset in the width direction.

[0082] Then, the output of the A-LIX layer is obtained using bilinear interpolation based on the following formulas (2) to (4). .

[0083] elements The following formula (2) must be satisfied:

[0084] (2); (3); (4);

[0085] in, The feature map representing the input elements, Represents the coordinates in the channel direction Coordinates representing the height direction, Represents the coordinates in the width direction. This indicates rounding down. This indicates rounding up. During training, the A-LIX layer smooths the gradients of the convolutional layers, resulting in a better surgical operation model. While the A-LIX layer needs to smooth the gradients of the convolutional layers during training, it does not need to do so during testing. Therefore, the input to the A-LIX layer is equal to its output.

[0086] The following is combined Figure 5 The interactive diagram of the surgical operation model and the surgical environment illustrates the process of determining the target action and the reward value.

[0087] The process of determining the target action can be modeled as a Markov decision process, which can be represented by a six-tuple. It means that, among them, Representing the state space, Represents the observation space Represents the action space, Represents the state transition function. Represents the reward function, Indicates the attenuation coefficient. Represents the real number field and is a general mathematical set notation. At the first moment... The second image sample data and the first image sample data The encoded sample information obtained and the action sample information corresponding to the first image sample data. The input is fed into the policy estimation sub-model, which can output the predicted probability of selecting each action at the first time step, and select the action with the highest probability as the target action. ,in, .

[0088] In performing the target action The state of the surgical environment from by state transition probability Transform into .

[0089] In performing the target action It is possible to determine the reward value of performing this target action. , Determine the return value. The specific process can be as follows:

[0090] Perform the target action to obtain the position of the device in the blood vessel at a first moment;

[0091] The reward value at the first moment is determined based on whether the position deviates from the target path, whether the contact force of the device is greater than or equal to a preset threshold, and the difference between the position at the first moment and the position at the second moment when the device moves from the position at the first moment to the position at the second moment.

[0092] For example, the reward value of the target action can be determined based on three factors: whether the device deviates from the correct path after performing the target action at the first moment; whether the device is getting closer to the endpoint while on the correct path; and whether the contact force of the device is within a preset range. Specifically, this can be achieved by:

[0093] After performing the target action, the position of the device in the blood vessel at the first moment can be determined.

[0094] Furthermore, the algorithm can determine whether the device has deviated from the correct path based on its position in the blood vessel at the first moment. If the device deviates from the correct path and enters an incorrect blood vessel branch, a preset penalty will be applied; conversely, if the device leaves the incorrect blood vessel branch and returns to the correct path, a preset reward will be applied. The correct path is the shortest path from the device's starting position in the blood vessel to its ending position, automatically generated by Dijkstra's algorithm. For example, if the device deviates from the correct path and enters an incorrect blood vessel branch after performing the target action at the first moment, the preset penalty value can be -2, and the reward value will also be -2. If the device performs the target action and enters the correct blood vessel at the first moment, the preset reward value can be +2, and the reward value will also be +2.

[0095] When the device moves from its position at the first moment to its position at the second moment, based on the correct path, it can be considered to be moving forward or backward on the correct path. The reward value at the first moment can be determined based on the difference between the device's position at the first moment and its position at the second moment. For example, if the distance between the device's position and the target position is 9 pixels at the first moment and 10 pixels at the second moment, the difference between the device's position at the second moment and its position at the first moment relative to the target position is 1 pixel, so the reward can be +1, and the reward value is also +1. As another example, if the distance between the device's position and the target position is 9 pixels at the first moment and 8 pixels at the second moment, the difference between the device's position at the second moment and its position at the first moment relative to the target position is -1 pixel, so the penalty value can be -1, and the reward value is also -1.

[0096] Based on the above-described determination of the reward value, the reward value can be further determined by considering whether the contact force of the device is greater than or equal to a preset threshold. The motor current can be determined based on the contact force of the device. When the motor current exceeds the threshold, it is determined that the contact force of the device exceeds the preset threshold, and a penalty is applied, for example, a penalty value of -1 can be set. When the contact force of the device does not exceed the threshold, neither a reward nor a penalty is applied. The reward and penalty values ​​can be set based on actual needs, and this invention does not impose any limitations on this.

[0097] It should be noted that during model training, the reward value for each target action can be determined, resulting in a cumulative reward value for the target surgical operation model obtained after training. The largest, of which, This means that as t increases, Decrease, as t decreases Increase, 0 < <1.

[0098] The control device for vascular interventional surgery provided in this invention trains the model using an offline reinforcement learning method, maximizing the cumulative reward value of the trained target surgical operation model. Compared to existing technologies that merely imitate learning methods to train the model, the model trained in this invention has higher accuracy. By configuring the target surgical operation model within a vascular interventional surgery robot, the accuracy of the robot's control instruments moving within the blood vessel is significantly improved.

[0099] In one embodiment, the model parameters of the initial surgical operation model are updated based on the reward value and the function estimate to obtain the target surgical operation model, which can be achieved in the following way:

[0100] Based on the reward value, the estimated value of the function, the estimated value of the objective function, and the advantage of the sample data relative to the agent's policy, the function estimation loss function is determined;

[0101] The policy mimicry loss function is determined based on the agent's policy and the sample data.

[0102] Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain the target surgical operation model.

[0103] For example, the function estimation loss function can be determined based on the following formula (5). :

[0104] (5);

[0105] in Represents the expectation on the sample data For hyperparameters, express The middle corresponds to the action elements, for , This indicates the advantage of sample data over agent strategies. This is an estimate of the function.

[0106] The estimated value of the objective function can be expressed by the following formula (6):

[0107] (6);

[0108] Estimation branch of the objective function parameters It is a branch of the function estimate. The average of the exponential.

[0109] exist After each update, The update is performed using the following formula (7):

[0110] (7);

[0111] in It represents the exponential average speed.

[0112] The policy imitation loss function can be determined based on the following formula (8). :

[0113] (8);

[0114] Once the function estimation loss function and the policy imitation loss function are obtained, gradient descent can be used to optimize them, update the model parameters of the initial surgical operation model, and obtain the target surgical operation model.

[0115] The control device for vascular interventional surgery provided in this embodiment of the invention optimizes the model parameters of the initial surgical operation model based on the function estimation loss function and the policy imitation loss function. Since the function estimation loss function can reflect the error between the estimated value of the function and the estimated value of the objective function, as well as the advantage of the sample data relative to the agent's policy, and the policy imitation loss function can reflect the relationship between the agent's policies at different times, the accuracy of the optimized model is higher.

[0116] In one embodiment, the model parameters of the initial surgical operation model are optimized based on the function estimation loss function and the policy imitation loss function to obtain the target surgical operation model. This can be achieved in the following ways:

[0117] Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain an imitation-based surgical operation model;

[0118] Based on the agent's policy and the estimated value of the objective function, the policy optimization loss function is determined;

[0119] Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

[0120] For example, the model parameters of the initial surgical operation model are optimized based on the function estimation loss function and the policy imitation loss function. The overall optimization idea is to optimize the imitation policy, so as to obtain a surgical operation model based on imitation.

[0121] Based on the obtained imitation-based surgical operation model, the weight of each sample data is initialized to... At this point, each sample data has the same weight. After randomly sampling the sample data, the loss function can be optimized using a strategy, and the loss function can be estimated using the function described in the above embodiment. The model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

[0122] The loss function for function estimation has been described in detail in the above embodiments and will not be repeated here.

[0123] The policy optimization loss function can be determined based on the following formula (9). :

[0124] (9);

[0125] in, Represents entropy, This indicates a non-negative parameter.

[0126] It can be automatically adjusted by the following formula (10):

[0127] (10);

[0128] in, for loss function, hyperparameters It is the target entropy.

[0129] Furthermore, during training, Lagrange relaxation can be used on the A-LIX layer in the encoder to automatically adjust the translation range. This is to ensure that the smoothness of the gradients in the convolutional layers remains within an appropriate range. For the gradients output by the A-LIX layer... The smoothness can be expressed by the modified normalized discontinuity score (11). To measure:

[0130] (11);

[0131] in Used to measure the smoothness of gradients. H represents the number of channels in the convolutional layer, and H represents the height of the feature map. Indicates the width of the feature map. yes The expected squared local discontinuity can be expressed by the following formula (12):

[0132] (12);

[0133] in, The gradient can be represented using a matrix. The direction of the matrix is ​​when The smaller the size, the more it means The smoother it is.

[0134] Additionally, the translation range The following formula (13) can be used for updating:

[0135] (13);

[0136] in, for loss function, hyperparameters yes The target value.

[0137] The control device for vascular interventional surgery in this embodiment of the invention further optimizes the model parameters of the imitation-based surgical operation model based on the model optimized by the imitation parameters, based on the function estimation loss function and the strategy optimization loss function, so that the accuracy of the target surgical operation model finally trained is higher, and the accuracy of the instrument movement in the blood vessel is further improved.

[0138] In one embodiment, the model parameters of the imitation-based surgical operation model are optimized based on the strategy optimization loss function and the function estimation loss function to obtain the target surgical operation model. This can be achieved in the following ways:

[0139] The sampling probability of the corresponding sample data is determined based on the weight of each sample data.

[0140] The target sample data is determined based on the sampling probability of each of the sample data.

[0141] The target sample data is input into the imitation-based surgical operation model. Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

[0142] For example, the sampling probability of the i-th sample data can be determined based on the following formula (14):

[0143] (14);

[0144] in Indicates the first The probability that a sample data point will be sampled. Represents the total number of samples. Indicates the first The weights of each sample data.

[0145] The sampling probability of each sample data is proportional to its weight. The larger the weight of the i-th sample data, the greater the probability that the i-th sample data will be sampled. Therefore, when obtaining the sampling probability of each sample data, we can compare the sampling probability of each sample data with the preset probability, and determine the sample data corresponding to the sampling probability greater than the preset probability as the target sample data. Then, the target sample data is used as the training data for this round of training of the imitation-based surgical operation model. Based on the policy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized.

[0146] In the control device for vascular interventional surgery in this embodiment of the invention, when optimizing the model parameters of the imitation-based surgical operation model based on the strategy optimization loss function and the function estimation loss function, the sampling probability of the corresponding sample data can be updated based on the weight of the sample data, so that the training data input into the imitation-based surgical operation model in each round is more accurate.

[0147] In one embodiment, the control device for the vascular interventional procedure can also be implemented in the following way:

[0148] The update module is used to update the weights of the target sample data based on the reward value, the estimated value of the objective function, and the estimated value of the function after optimizing the model parameters of the imitation-based surgical operation model in each round.

[0149] For example, Figure 6 This is a schematic diagram illustrating the process of optimizing the model parameters of a surgical operation model based on imitation, as provided by the present invention. After optimizing the model parameters of the surgical operation model based on imitation in each round, the weights of each sampled target data need to be updated, and the sampling probability of the corresponding target data is updated based on the updated weights. Then, a new round of sample data is determined based on the updated sampling probability. Using the new round of sample data, the model parameters of the surgical operation model based on imitation are further optimized based on the strategy optimization loss function and the function estimation loss function. This process is repeated until the convergence condition is met, and the target surgical operation model is obtained.

[0150] Specifically, the weights of the target sample data can be updated based on the following formula (15):

[0151] (15);

[0152] in, It is the first Temporal difference error of target sample data, hyperparameter Used to control the update speed of weights, and Used to limit the range of weights to prevent them from being too small or too large.

[0153] The control device for vascular interventional surgery in this embodiment of the invention updates the sampling weight of each sample data based on the reward value, the estimated value of the objective function, and the function estimate, so that the sample data input in each round is more accurate, and further improves the accuracy of the target surgical operation model finally trained.

[0154] The control method for vascular interventional surgery provided by the present invention is described below. The control method for vascular interventional surgery described below can be referred to in correspondence with the control device for vascular interventional surgery described above.

[0155] Figure 7 This is a flowchart illustrating the control method for vascular interventional surgery provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the control method for this vascular interventional procedure includes the following steps:

[0156] Step 701: Obtain the first image data from the previous moment, the second image data from the current moment, and the first action information from the previous moment; both the first image data and the second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery.

[0157] Step 702: Input the first image data, the second image data, and the first action information into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data, based on an offline reinforcement learning method; the first image sample data is the data collected at the moment before the second image sample data.

[0158] Step 703: Determine the target control command based on the probability of selecting each of the actions.

[0159] Step 704: Control the movement of the device in the blood vessel based on the target control command.

[0160] The control method for vascular interventional surgery provided by this invention inputs the first image data from the previous moment, the second image data from the current moment, and the first action information from the previous moment into a target surgical operation model. This target surgical operation model can output the probability of instrument actions being selected, thereby determining the target control command corresponding to the instrument's action, and then controlling the movement of the instrument in the blood vessel according to the target control command. Since this target surgical operation model is trained using an offline reinforcement learning method, which optimizes model parameters based on a reward mechanism and does not depend on the quality of the physician's operation examples, the accuracy of the trained target surgical operation model is improved, further enhancing the accuracy of instrument movement in the blood vessel. Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a control method for vascular interventional surgery. This method includes: acquiring first image data from the previous moment, second image data from the current moment, and first action information from the previous moment; both the first image data and the second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery.

[0161] The first image data, the second image data, and the first action information are input into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data, based on an offline reinforcement learning method; the first image sample data is the data collected at the moment preceding the second image sample data.

[0162] The target control command is determined based on the probability of selecting each of the actions;

[0163] The device is controlled to move within the blood vessel based on the target control command.

[0164] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the control method for vascular interventional surgery provided by the above methods, the method including: acquiring first image data at the previous moment, second image data at the current moment, and first action information at the previous moment; the first image data and the second image data both include vascular image data and instrument image data corresponding to the vascular interventional surgery;

[0166] The first image data, the second image data, and the first action information are input into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data, based on an offline reinforcement learning method; the first image sample data is the data collected at the moment preceding the second image sample data.

[0167] The target control command is determined based on the probability of selecting each of the actions;

[0168] The device is controlled to move within the blood vessel based on the target control command.

[0169] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a control method for performing the vascular interventional surgery provided by the above methods. The method includes: acquiring first image data at a previous moment, second image data at a current moment, and first action information at the previous moment; both the first image data and the second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery.

[0170] The first image data, the second image data, and the first action information are input into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data, based on an offline reinforcement learning method; the first image sample data is the data collected at the moment preceding the second image sample data.

[0171] The target control command is determined based on the probability of selecting each of the actions;

[0172] The device is controlled to move within the blood vessel based on the target control command.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control device for vascular interventional surgery, characterized in that, The device includes: The acquisition module is used to acquire the first image data of the previous moment, the second image data of the current moment, and the first action information of the previous moment; both the first image data and the second image data include vascular image data and instrument image data corresponding to the vascular interventional surgery; The prediction module is used to input the first image data, the second image data, and the first action information into the target surgical operation model to obtain the probability of selecting each action at the current moment output by the target surgical operation model; the target surgical operation model is trained based on an offline reinforcement learning method using the first image sample data, the second image sample data, and the action sample information corresponding to the first image sample data; the first image sample data is the data collected at the moment preceding the second image sample data. A determination module is used to determine a target control command based on the probability of selecting each of the actions; The control module is used to control the movement of the device in the blood vessel based on the target control command; The target surgical operation model was trained in the following manner: The first image sample data and the second image sample data in the sample data are input into the encoder of the initial surgical operation model to obtain the encoded information sample output by the encoder; The encoded information samples and the action sample information in the sample data are input into the strategy estimation sub-model and function estimation sub-model of the initial surgical operation model to obtain the predicted probability of selecting each action at the first moment output by the strategy estimation sub-model and the function estimate value output by the function estimation sub-model; the first moment is the moment corresponding to the acquisition of the second image sample data; The target action is determined based on the predicted probability of each action selected at the first moment, and the reward value at the first moment is determined based on the target action. The model parameters of the initial surgical operation model are updated based on the reward value and the function estimate to obtain the target surgical operation model; the target surgical operation model corresponds to the maximum cumulative reward value. The step of updating the model parameters of the initial surgical operation model based on the reward value and the function estimate to obtain the target surgical operation model includes: Based on the reward value, the estimated value of the function, the estimated value of the objective function, and the advantage of the sample data relative to the agent's policy, the function estimation loss function is determined; The policy mimicry loss function is determined based on the agent's policy and the sample data. Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain the target surgical operation model.

2. The control device for vascular interventional surgery according to claim 1, characterized in that, The optimization of the model parameters of the initial surgical operation model based on the estimated loss function and the policy imitation loss function to obtain the target surgical operation model includes: Based on the function estimation loss function and the policy imitation loss function, the model parameters of the initial surgical operation model are optimized to obtain an imitation-based surgical operation model; Based on the agent's policy and the estimated value of the objective function, the policy optimization loss function is determined; Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

3. The control device for vascular interventional surgery according to claim 1, characterized in that, Determining the reward value at the first moment based on the target action includes: Perform the target action to obtain the position of the device in the blood vessel at a first moment; The reward value at the first moment is determined based on whether the position deviates from the target path, whether the contact force of the device is greater than or equal to a preset threshold, and the difference between the position at the first moment and the position at the second moment when the device moves from the position at the first moment to the position at the second moment.

4. The control device for vascular interventional surgery according to claim 2, characterized in that, The step of optimizing the model parameters of the imitation-based surgical operation model based on the optimized loss function and the estimated loss function to obtain the target surgical operation model includes: The sampling probability of the corresponding sample data is determined based on the weight of each sample data. The target sample data is determined based on the sampling probability of each of the sample data. The target sample data is input into the imitation-based surgical operation model. Based on the strategy optimization loss function and the function estimation loss function, the model parameters of the imitation-based surgical operation model are optimized to obtain the target surgical operation model.

5. The control device for vascular interventional surgery according to claim 4, characterized in that, The device further includes: The update module is used to update the weights of the target sample data based on the reward value, the estimated value of the objective function, and the estimated value of the function after optimizing the model parameters of the imitation-based surgical operation model in each round.

Images