Vascular interventional procedure action perception and learning system, method, and storage medium

Abstract

The application discloses a kind of vascular intervention operation action sensing and learning system, method and computer storage medium, including operation gesture acquisition unit, operation gesture analysis unit, intervention instrument accurate delivery unit;The operation gesture of vascular intervention operation of operator is sensed using multiple sensors, data cleaning is realized by Gaussian clustering, the learning of machine on time sequence to doctor gesture and corresponding effect is realized using deep learning model CNN-LSTM, and the autonomous operation of vascular intervention robot is realized according to relevant prediction results.The operation action of person can be converted to the action of machine autonomous operation in the application, and the machine has corresponding action learning function.Through the autonomous learning of surgical robot operation, the surgical robot gradually has the ability of independent work, and autonomously executes surgical tasks, so as to reduce various surgical risks caused by human factors.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent medical instruments, and particularly relates to a blood vessel intervention operation action sensing and learning system, method and storage medium. BACKGROUND

[0002] The blood vessel intervention operation is a surgical method for entering the body through puncture of a blood vessel, a guide wire, a catheter and other instruments to diagnose and treat lesions in the blood vessel. The blood vessel intervention operation has the advantages of accurate operation, short operation time, small operation trauma, short postoperative recovery time, and light pain suffered by patients, and gradually becomes one of the main means for treating cardiovascular diseases.

[0003] However, the traditional blood vessel intervention operation requires a doctor to manually send a catheter, a guide wire and a stent and other instruments into the body of a patient. During the operation, the doctor needs to wear a lead clothing anti-radiation garment weighing tens of kilograms, the physical strength decreases rapidly, the attention and stability are reduced, the operation precision is reduced, and accidents such as blood vessel puncture damage rupture caused by improper pushing force are easily caused, which causes danger to the patient's life. On the other hand, long-term radiation can also harm the health of the doctor.

[0004] In order to alleviate the above problems, people have developed many blood vessel intervention operation robots. At present, these operation robots still need a cardiologist to operate the equipment for remote operation. Most of the existing operation methods do not conform to the traditional experience and feeling of the intervention operation doctor, and the traditional operation gestures and experience of the doctor cannot be well used in the operation and use of the blood vessel intervention operation robot.

[0005] In order to better enable experienced doctors to play accumulated intervention operation experience, and to effectively realize the automatic operation of the blood vessel intervention operation robot, a blood vessel intervention operation action sensing and learning system is in urgent need. Not only can the gestures of the operator be accurately captured and stored, but also the computer can learn the gestures and operations of the operator through computer processing, and then be reproduced in the autonomous operation of the surgical robot, so as to enable the robot operation with human expert wisdom and improve the effect and quality of the robot operation.

[0006] Through the autonomous learning of the surgical robot for the operation, the surgical robot gradually has the ability to work independently, thereby autonomously performing the operation task, reducing the burden of the doctor, and being able to reduce the human factors in the operation process, thereby reducing the risk of operation failure. At the same time, the technology can also be used in the training of the intervention operation to give the students the operation feeling of the master's hand-in-hand operation, and improve the learning effect and efficiency.

[0007] At present, there is little research on the technology of surgical robot sensing and learning the gestures of the operator. The Chinese patent with the publication number CN201310359227.0 provides a data acquisition device of a brain surgery simulation training system. The invention can acquire the real-time position and rotation information of a surgical knife, and update the same, providing a data source for displaying the real-time information of the surgical knife in the simulation surgery software, and truly simulating the surgery training environment to help doctors realize the brain surgery simulation training. However, the invention cannot realize the conversion of human operation actions to machine autonomous operation actions, and the machine does not have a corresponding action learning function; and the system is not suitable for vascular interventional surgery. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a vascular interventional surgery action sensing and learning system, which solves the problem that the autonomous operation of the interventional surgery robot does not conform to the conventional habits of doctors in the prior art.

[0009] The present application adopts the following technical solutions to solve the above technical problems:

[0010] The vascular interventional surgery action sensing and learning system comprises an operation gesture acquisition unit, an operation gesture analysis unit and an interventional instrument precise delivery unit. The operation gesture acquisition unit comprises a plurality of sensors arranged on a surgical instrument, which are used to acquire physical quantity information related to the gesture actions of an operator. According to the physical quantity information related to the gesture actions, a plurality of “gesture-effect” pairs are defined and transmitted to the operation gesture analysis unit.

[0011] The operation gesture analysis unit evaluates and cleanses all “gesture-effect” pairs, filters out abnormal values of the “gesture-effect” pairs, retains valid “gesture-effect” pairs, and constructs a sample training set and a test set. The sample training set is used as the input of a CNN-LSTM model to learn the operation of the interventional instrument, acquire the time sequence actions of the interventional instrument driving, and train on the test set to acquire model evaluation indexes.

[0012] The interventional instrument precise delivery system autonomously drives and operates the interventional instrument according to the trained time sequence actions and in combination with real-time feedback information.

[0013] The sensors arranged on the operation end of the interventional surgery instrument include but are not limited to six-axis sensors, tension sensors, pressure sensors and positioning coils.

[0014] An electromagnetic positioning system is arranged on the driving end of the interventional surgery instrument.

[0015] The vascular interventional surgery action sensing and learning method comprises the following steps:

[0016] Step 1, the application operation gesture acquisition unit acquires the operation action of the operator, and decomposes the action into relevant physical quantity information, defines a plurality of "gesture-effect" pairs according to the received action-related physical quantity information, constructs a "gesture-effect" pair database, and sends to the surgical analysis unit;

[0017] Step 2, all "gesture-effect" pairs are evaluated and data cleaning, and abnormal values of "gesture-effect" pairs are filtered out, and valid "gesture-effect" pairs are retained, and a sample training set and a test set are constructed;

[0018] Step 3, the sample training set is taken as the input of the CNN-LSTM model, the learning of the interventional instrument operation is carried out, the time sequence action of the interventional instrument driving is acquired, and the training on the test set is carried out, the model evaluation index is acquired, and the prediction model is acquired according to the model evaluation index;

[0019] Step 4, the interventional instrument precise delivery unit realizes the autonomous driving operation of the interventional instrument according to the prediction information of the prediction model and the real-time feedback information.

[0020] The gesture acquisition unit decomposes the acquired action into translation delivery and twisting rotation; and records physical quantities including but not limited to hand twisting angular velocity of the catheter, delivery force, forward and backward movement speed, time, interval, and pressure; and in the same time domain, the corresponding rotation angular velocity and pose of the interventional instrument, forward and backward movement speed, time, interval, and action intention in special positions.

[0021] The gesture analysis unit evaluates the effect of each gesture by the Gaussian clustering method, and the initial data is given by experienced interventional physicians; then, data cleaning is performed on the clustered gestures to obtain effective "gesture-effect" pairs, which are stored in the computer and construct a "gesture-effect" pair database.

[0022] The Gaussian clustering method is to apply a Gaussian mixture model to form a mixed probability model by weighting a plurality of single Gaussian distributions by a certain weight, and the specific process is as follows:

[0023] S1. For K initial Gaussian distributions, calculate the probability of each sample belonging to each category under the given multi-dimensional Gaussian distribution;

[0024] S2. Recalculate the more optimal Gaussian parameters according to the probability;

[0025] S3. Then continuously iterate the above steps until the likelihood function changes is no longer obvious, that is, the difference between adjacent two iterations is less than 0.0001, or the maximum number of iterations is reached.

[0026] The CNN-LSTM model is applied to realize learning of driving operation of an interventional instrument on a time sequence action, and the learning method of the interventional instrument operation is as follows:

[0027] Step a: the CNN network extracts features of a "gesture-effect" pair in each frame action; and the LSTM network extracts features of time sequence changes of the "gesture-effect" pair;

[0028] Step b: data acquired in step a is input into the CNN-LSTM model for training to acquire a "reasonable speed" when the surgery reaches a certain stage.

[0029] The specific process of step 4 is as follows:

[0030] S4.1: an electromagnetic positioning system is arranged on a driving end of the interventional instrument, and real-time tracking of position information is realized according to position information fed back by the electromagnetic positioning system;

[0031] S4.2: the system calculates corresponding position information according to changes of the position information on the time sequence;

[0032] S4.3: the system monitors and adjusts real-time movement according to the reasonable speed of the surgery calculated by the deep learning model when the surgery reaches the state.

[0033] A computer storage medium stores computer instructions, and the computer instructions are used to execute all or part of the steps of the method when called.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] 1: The present application can convert human operation actions to machine autonomous operation actions, the machine has a corresponding action learning function, the surgery robot learns the surgery operation autonomously to make the surgery robot gradually have the ability of independent work and autonomously execute a surgery task, thereby reducing various surgery risks caused by human factors.

[0036] 2: The present application uses a Gaussian clustering method to realize evaluation of effects generated by each gesture, has a fast convergence speed and good effect on a large-scale data set.

[0037] 3: The present application uses a CNN-LSTM model to effectively classify and learn actions on a time sequence. BRIEF DESCRIPTION OF DRAWINGS

[0038] Fig. 1 It is a system module composition schematic diagram of the present application.

[0039] Fig. 2 It is a method step flow schematic diagram of the present application.

[0040] Fig. 3 The schematic diagram of the neural network composition of the present application. DETAILED DESCRIPTION

[0041] The structure and working process of the present application will be further described below in combination with the drawings.

[0042] The blood vessel intervention operation action sensing and learning system comprises an operation gesture acquisition unit, an operation gesture analysis unit and an intervention instrument precise delivery unit; wherein the operation gesture acquisition unit comprises a plurality of sensors arranged on the surgical instrument, which are used to acquire physical quantity information related to the gesture action of the operator, define a plurality of "gesture-effect" pairs according to the physical quantity information related to the gesture action, and transmit them to the operation gesture analysis unit.

[0043] The operation gesture analysis unit evaluates and cleanses all "gesture-effect" pairs, filters out abnormal values of "gesture-effect" pairs, retains valid "gesture-effect" pairs, and constructs a sample training set and a test set; the sample training set is used as the input of the CNN-LSTM model to learn the operation of the intervention instrument, acquire the time sequence action of the intervention instrument driving, and train on the test set to acquire model evaluation indexes.

[0044] The intervention instrument precise delivery system performs autonomous driving operation on the intervention instrument according to the trained time sequence action combined with real-time feedback information.

[0045] Specific embodiments, as shown in Figs. 1 to 3

[0046] The blood vessel intervention operation action sensing and learning system comprises an operation gesture acquisition unit, an operation gesture analysis unit and an intervention instrument precise delivery unit; wherein the operation gesture acquisition unit comprises a plurality of sensors arranged on the surgical instrument, which are used to acquire physical quantity information related to the gesture action of the operator, define a plurality of "gesture-effect" pairs according to the physical quantity information related to the gesture action, and transmit them to the operation gesture analysis unit.

[0047] ​The operation gesture analysis unit evaluates and cleanses data of all "gesture-effect" pairs, filters out "gesture-effect" pair outliers, retains valid "gesture-effect" pairs, and constructs a sample training set and a test set; the sample training set is used as the input of the CNN-LSTM model to learn the operation of the interventional instrument, obtain the timing action driven by the interventional instrument, and train on the test set to obtain model evaluation indicators; the evaluation of the effect of each gesture is realized by the Gaussian clustering method, and the initial data will be given by experienced interventional physicians; then, the gestures after clustering are cleaned up to obtain good gesture-effect pairs, which are stored in the computer and provided for subsequent operations.

[0048] The interventional instrument precise delivery system autonomously drives the interventional instrument according to the trained timing action and in combination with real-time feedback information. An electromagnetic positioning system is arranged on the driving end of the interventional surgery instrument.

[0049] In order to effectively perceive the physical quantities related to the operation action of the operator, in this embodiment, a corresponding six-axis sensor and a corresponding electromagnetic positioning system are arranged on the driving end of the interventional instrument and the operator's hand operation end to realize the measurement of physical quantities such as rotation and displacement in each direction. Meanwhile, a tension and compression force sensor is arranged to realize the relationship between the tension of the doctor before and after delivery and the displacement. In order to realize force detection, a DAYSENSOR DYHW-120 type tension and compression force sensor is selected; the electromagnetic positioning system is an NDI Aurora V3 series medical magnetic field generator and a matching positioning coil.

[0050] The specific device model given above is only an example to illustrate the scheme and is not limiting, and those skilled in the art can make adaptive selection and application according to specific needs, as long as the functions described can be realized, which can be replaced by devices.

[0051] The blood vessel interventional surgery action perception and learning method comprises the following steps:

[0052] Step 1, an operation gesture acquisition unit is applied to acquire the operation action of the operator, and the action is decomposed into related physical quantity information; a plurality of "gesture-effect" pairs are defined according to the received action-related physical quantity information, a "gesture-effect" pair database is constructed, and is sent to a surgery analysis unit;

[0053] Step 2, all "gesture-effect" pairs are evaluated and cleaned up, "gesture-effect" pair outliers are filtered out, valid "gesture-effect" pairs are retained, and a sample training set and a test set are constructed;

[0054] Step 3, taking the sample training set as the input of the CNN-LSTM model, learning the operation of the interventional instrument, obtaining the time sequence action of the interventional instrument driving, and training on the test set to obtain the model evaluation index, and obtaining the prediction model according to the model evaluation index;

[0055] Step 4, the interventional instrument precision delivery unit realizes the autonomous driving operation of the interventional instrument according to the prediction information of the prediction model and the real-time feedback information. Specific embodiments

[0057] The blood vessel intervention operation action sensing and learning method comprises the following steps:

[0058] Step 1, the operation gesture acquisition unit acquires the operation action of the operator, and decomposes the action into related physical quantity information, defines a plurality of "gesture-effect" pairs according to the received action related physical quantity information, constructs a "gesture-effect" pair database, and sends it to the operation analysis unit; the gesture acquisition unit decomposes the acquired action into translation delivery and twisting rotation; and records physical quantities including but not limited to hand twisting angular velocity of the catheter, delivery force, forward and backward movement speed, time, interval, pressure, and physical quantities of the interventional instrument corresponding to the rotation angular velocity and pose, forward and backward movement speed, time, interval, and action intention at a special position in the same time domain.

[0059] The effects of each gesture are evaluated by a Gaussian clustering method, and the initial data will be given by experienced interventional physicians; then, the gestures after clustering are data cleaned to obtain effective "gesture-effect" pairs, which are stored in the computer to construct a "gesture-effect" pair database.

[0060] The Gaussian Mixture Model (Gaussian Mixture Model) is a mixed probability model obtained by weighting a plurality of single Gaussian distributions by a certain weight, so that the model capacity is larger, and more complex sampling or fitting of more complex distribution is generated.

[0061] GMM assumes a probability density function subject to Gaussian distribution for the feature distribution under each category:

[0062]

[0063] P(x|c k )~N(μ k ,σ k )

[0064] And the data may be mixed by multiple categories, so the probability density function of the features in the data can be represented by a combination of multiple Gaussian distributions:

[0065]

[0066] where π k is the class distribution probability, which can also be regarded as the weight coefficient of each Gaussian distribution function, also called the mixture coefficient, which satisfies

[0067]

[0068] Under multi-dimensional data, a multi-dimensional Gaussian distribution needs to be generated for each class:

[0069]

[0070] The Gaussian clustering method is to apply a Gaussian mixture model to form a mixed probability model by weighting multiple single Gaussian distributions by a certain weight, and the specific process is as follows:

[0071] S1. For K initial Gaussian distributions, calculate the probability of each sample belonging to each class under the given multi-dimensional Gaussian distribution:

[0072]

[0073] S2. Recalculate the more optimal Gaussian parameters according to the probability:

[0074]

[0075] S3. Then continuously iterate the above steps until the likelihood function changes are no longer obvious, i.e., the difference between adjacent two iterations is less than 0.0001, or the maximum number of iterations is reached.

[0076] Step 2, evaluate and clean all "gesture-effect" pairs, filter out "gesture-effect" pair outliers, retain valid "gesture-effect" pairs, and construct a sample training set and test set; for all samples in the data, the probability (likelihood function) of its occurrence is:

[0077]

[0078] The logarithmic likelihood function is:

[0079]

[0080] Step 3, use the sample training set as the input of the CNN-LSTM model to learn the interventional instrument operation, obtain the timing action of the interventional instrument driving, and train on the test set to obtain the model evaluation index, and according to the model evaluation index, obtain the prediction model;

[0081] The CNN-LSTM model is a fusion model of a convolutional neural network (CNN) and a long short-term memory network (LSTM). Generally, the fusion of CNN and LSTM is often used for spatiotemporal modeling tasks.

[0082] The CNN has the feature of paying attention to the most obvious features in information, and is therefore widely used in feature extraction engineering. The LSTM has the feature of expanding in time sequence, and is widely used in information extraction in time series. Therefore, the combination of the two, CNN-LSTM, can well complete the extraction and analysis of time series actions.

[0083] The CNN-LSTM used in this embodiment has two hidden layers, each with 20 neurons, two convolutional layers and two pooling layers, 500 training iteration cycles, 64 samples per batch for network training, 64 filter numbers, and a convolution kernel size of 3, as shown in the accompanying Fig. 3 .

[0084] The specific learning method is as follows:

[0085] Step a, the CNN network extracts the features of the "gesture-effect" pair in each frame of action; the LSTM network extracts the features of the time sequence change of the "gesture-effect" pair;

[0086] Step b, the data obtained in step a is input into the CNN-LSTM model for training to obtain the "reasonable speed" when the surgery reaches a certain stage.

[0087] Step 4, the interventional instrument precise delivery unit realizes the autonomous driving operation of the interventional instrument according to the prediction information of the prediction model and the real-time feedback information. The specific process is as follows:

[0088] S4.1, an electromagnetic positioning system is arranged on the driving end of the interventional instrument, and the real-time tracking of the position information is realized according to the position information fed back by the electromagnetic positioning system;

[0089] S4.2, the system calculates the corresponding position information according to the change of the position information in time sequence;

[0090] S4.3, the system monitors and adjusts the real-time motion according to the reasonable speed of the surgery reaching the state calculated by the deep learning model.

[0091] The above functions, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Therefore, a computer storage medium is also disclosed, which stores computer instructions, and the computer instructions are used to execute all or part of the steps of the method when called.

[0092] Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0093] Compared with the prior art, the present application can convert human operation actions to machine autonomous operation actions, and the machine has a corresponding action learning function. Through autonomous learning of surgical operation by the surgical robot, the surgical robot gradually has the ability to work independently, and autonomously performs a surgical task, thereby reducing various surgical risks caused by human factors.

[0094] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover the non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] The above describes the preferred embodiments of the present application in detail. It should be understood that the present application is not limited to the specific implementation described above, and the equipment and structures not described in detail should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the present application or modify equivalent embodiments with the disclosed methods and technical content without departing from the scope of the present application, which does not affect the essential content of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiments based on the present application and the existing technology by those skilled in the art should be within the protection scope determined by the claims.

Claims

1. A vascular intervention procedure action sensing and learning system, characterized by: The method comprises an operation gesture acquisition unit, an operation gesture analysis unit and an interventional instrument precise delivery unit; the operation gesture acquisition unit comprises a plurality of sensors arranged on a surgical instrument, and is used to acquire physical quantity information related to gesture actions of an operator, define a plurality of "gesture-effect" pairs according to the physical quantity information related to the gesture actions, and transmit the "gesture-effect" pairs to the operation gesture analysis unit; The operation gesture analysis unit evaluates all "gesture-effect" pairs and performs data cleaning, filters out abnormal values of the "gesture-effect" pairs, retains valid "gesture-effect" pairs, and constructs a sample training set and a test set; the sample training set is used as an input of a CNN-LSTM model to learn operation of an interventional instrument, acquire time sequence actions of driving of the interventional instrument, and train on the test set to acquire a model evaluation index; The interventional instrument precise delivery system performs autonomous driving operation on the interventional instrument according to the trained time sequence actions and real-time feedback information; the method for action sensing and learning of a vascular interventional surgery comprises the following steps: Step 1: an operation gesture acquisition unit is applied to acquire operation actions of a surgeon, and the actions are decomposed into related physical quantity information; a plurality of "gesture-effect" pairs are defined according to the received physical quantity information related to the actions, a "gesture-effect" pair database is constructed, and the "gesture-effect" pairs are sent to a surgical analysis unit; the gesture acquisition unit decomposes the acquired actions into translational delivery and twisting rotation; and records physical quantities including but not limited to a twisting angular velocity of a hand on a catheter, a force size of delivery, a forward and backward movement speed, a time, an interval, a pressure size, a corresponding rotational angular velocity and a pose of an interventional instrument in the same time domain, a forward and backward movement speed, a time, an interval, and an action intention in a special position; the gesture analysis unit evaluates effects of each gesture by a Gaussian clustering method, initial data of which is given by an experienced interventional physician; then, data cleaning is performed on the clustered gestures to obtain valid "gesture-effect" pairs, which are stored in a computer and used to construct the "gesture-effect" pair database; the Gaussian clustering method is a method of applying a Gaussian mixture model to form a mixed probability model by weighting a plurality of single Gaussian distributions by a certain weight, and the specific process is as follows: S1. Calculate probabilities of each sample belonging to each category under a given multi-dimensional Gaussian distribution for K initial Gaussian distributions; S2. Recalculate more optimal Gaussian parameters according to the probabilities; S3. Then, the above steps are iterated constantly until a likelihood function changes no longer obviously, that is, a difference between adjacent two iterations is less than 0.0001, or a maximum iteration number is reached; Step 2: evaluate all "gesture-effect" pairs and perform data cleaning, filter out abnormal values of the "gesture-effect" pairs, retain valid "gesture-effect" pairs, and construct a sample training set and a test set; Step 3: use the sample training set as an input of a CNN-LSTM model to learn operation of an interventional instrument, acquire time sequence actions of driving of the interventional instrument, train on the test set to acquire a model evaluation index, and acquire a prediction model according to the model evaluation index; Step 4, precise delivery unit of interventional instrument, according to the prediction information of the prediction model and the real-time feedback information, realize the autonomous driving operation of the interventional instrument.

2. The vascular intervention procedure action sensing and learning system of claim 1, wherein: The sensors arranged on the operation end of the interventional surgical instrument include but are not limited to six-axis sensors, tension sensors, pressure sensors, and positioning coils.

3. The vascular intervention procedure action sensing and learning system of claim 2, wherein: An electromagnetic positioning system is arranged on the driving end of the interventional surgical instrument.

4. The method of action sensing and learning for vascular interventions of claim 1, wherein: The CNN-LSTM model is applied to realize the learning of the driving operation of the interventional instrument on the time sequence action, and the learning method of the interventional instrument operation is as follows: Step a, the CNN network extracts the features of the "gesture-effect" pair in each frame of action; the LSTM network extracts the features of the time sequence change of the "gesture-effect" pair; Step b, the data obtained in step a is input into the CNN-LSTM model for training, and the "reasonable speed" when the operation reaches a certain stage is obtained.

5. The method of action sensing and learning for vascular interventions of claim 4, wherein: The specific process of step 4 is as follows: S4.1, an electromagnetic positioning system is arranged on the driving end of the interventional instrument, and the real-time tracking of the position information is realized according to the position information fed back by the electromagnetic positioning system; S4.2, the system calculates the corresponding position information according to the change of the position information on the time sequence; S4.3, the system monitors and adjusts the real-time motion according to the reasonable speed calculated by the deep learning model when the operation reaches a certain stage.

6. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to execute all or part of the steps of the method of any one of claims 1 to 5 when called.

Images