Information prompting method and device of interventional operation robot and storage medium

Abstract

The invention relates to the field of medical equipment, and provides an information prompting method and device of an interventional operation robot and a storage medium, the method comprises the steps of obtaining a to-be-recognized target image, and recognizing object features of a target object in the target image, the target object at least comprising a multi-channel valve body and / or a medical instrument located in the multi-channel valve body; inputting the object features into a pre-trained neural network model, and outputting a valve body state of the multi-channel valve body and / or an instrument position state of the medical instrument in the multi-channel valve body; if it is determined that blood backflow occurs in the multi-channel valve body according to the state of the valve body, first information prompt related operation is executed; and / or if it is determined that the medical instrument is withdrawn to the target position according to the instrument position state, second information prompt related operation is executed. The manpower demand in the interventional operation process is reduced, and the safety and intelligence of the interventional operation robot are improved.

Description

Technical Field

[0001] The present application relates to the field of medical equipment, and in particular to an information prompting method, device and storage medium for an interventional surgical robot. Background Art

[0002] The interventional surgical robot can obtain the operator's operating instructions at the master end, and control the medical device from the slave end according to the operating instructions, so that the operator can perform vascular interventional surgery in a separate room, showing many advantages of robotic surgery, such as radiation protection, precise operation, doctor-patient isolation, etc. Although robotic surgery has many advantages, the introduction of robots also adds many uncertain factors.

[0003] For example, unexpected instrument movement, deformation and detachment of sheath connectors due to external forces, instrument bleeding, abnormal infusion pressure, etc., may lead to abnormalities such as leakage and blood return at the T-valve, affecting the safety of interventional surgical operations. In addition, during the operation of existing interventional surgical robots, manpower is still needed to continuously monitor the position of the robot's medical devices in the operating room inside and outside the valve. Taking the guide wire as an example of a medical device, before injecting contrast agent into the multi-channel valve body, the guide wire needs to be withdrawn to prevent the guide wire from affecting the injection of the contrast agent. However, if the guide wire is completely withdrawn from the T-valve, it may be necessary to reinsert the guide wire into the multi-channel valve body later, resulting in cumbersome operation, greatly extended operation time, and high labor costs.

[0004] Therefore, there is an urgent need for an interventional robot that can monitor whether there is blood return in the T-valve and / or the position status of medical devices such as guidewires and output prompts based on the monitoring results, so as to improve the safety and work efficiency of the interventional surgical robot. Summary of the invention

[0005] The main purpose of the present application is to provide an information prompting method, device and storage medium for an interventional surgical robot, aiming to enable the interventional surgical robot to monitor and prompt whether there is blood return in the valve body and / or whether the medical device has been withdrawn to the target position, thereby improving the safety and work efficiency of the interventional surgical robot.

[0006] In a first aspect, the present application provides an information prompting method for an interventional surgery robot, the information prompting method for an interventional surgery robot comprising the following steps:

[0007] Acquire a target image to be identified, and identify object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body;

[0008] Inputting the object features into a pre-trained neural network model to output at least a valve body state of the multi-channel valve body and / or an instrument position state of the medical instrument in the multi-channel valve body;

[0009] If it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body, executing a first information prompt related operation;

[0010] If it is determined according to the valve body state that no blood reflux occurs in the multi-channel valve body, then the first information prompt related operation is not performed;

[0011] and / or,

[0012] If it is determined according to the device position state that the medical device has retreated to the target position, executing a second information prompt related operation;

[0013] If it is determined according to the device position status that the medical device has not retreated to the target position, the second information prompt related operation is not performed.

[0014] In a second aspect, the present application further provides an information prompting device for an interventional surgery robot, the information prompting device comprising:

[0015] An image recognition module, used to acquire a target image to be recognized and recognize object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body;

[0016] A state information acquisition module, used for inputting the object features into a pre-trained neural network model, and outputting at least the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body;

[0017] An information prompt module is used to execute a first information prompt-related operation when it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body; not execute the first information prompt-related operation when it is determined according to the valve body state that blood reflux does not occur in the multi-channel valve body; and / or execute a second information prompt-related operation when it is determined according to the device position state that the medical device has retreated to the target position; and not execute the second information prompt-related operation when it is determined according to the device position state that the medical device has not retreated to the target position.

[0018] In a third aspect, the present application further provides an interventional surgery robot, comprising: an interventional surgery robot slave end, an interventional surgery robot master end, and a controller;

[0019] The interventional surgical robot slave end is provided with a valve body assembly mounting portion, the valve body assembly mounting portion is installed with a multi-channel valve body, the multi-channel valve body is used to accommodate medical equipment, and a camera device is also provided at a position opposite to the valve body assembly mounting portion and the multi-channel valve body, for acquiring a target image of the multi-channel valve body;

[0020] The interventional surgery robot master end is used to receive the user's operation instructions and control the interventional surgery robot slave end through the controller;

[0021] The controller is used to execute the information prompting method of the interventional surgery robot.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the information prompt method of the interventional surgical robot as described above is implemented.

[0023] The present application provides an information prompt method, device and storage medium for an interventional surgical robot. By identifying the valve body status in the target image and performing corresponding information prompt operations according to the identification results, the machine can make independent judgments when abnormalities such as blood leakage and blood return occur at the multi-channel valve body, and promptly prompt the doctor, thereby improving the safety of the operation. By identifying the position status of medical devices, prompt information is given when the medical devices retreat to the target position, so that medical devices such as guidewires and catheters do not need to leave the multi-channel valve body, eliminating the step of withdrawing and re-inserting the multi-channel valve body during the operation, improving the work efficiency of the interventional surgical robot, and eliminating the need for additional manpower to monitor the machine status in the operating room, further reducing the risk of radiation to doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of a flow chart of an information prompting method for an interventional surgery robot provided in one embodiment of the present application;

[0026] Figure 2 A structural diagram of an interventional surgery robot slave end provided in an embodiment of the present application;

[0027] Figure 3 A schematic block diagram of the structure of an interventional surgery robot slave terminal provided in one embodiment of the present application;

[0028] Figure 4a , Figure 4b A schematic diagram showing the comparison of valve body states in a target image provided by an embodiment of the present application;

[0029] Figure 5a , Figure 5b A schematic diagram for comparing the position states of instruments in a target image provided by an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a process for training a state recognition model provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of a module of an information prompting device for an interventional surgery robot provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0034] The embodiments of the present application provide an information prompting method, device and storage medium for an interventional surgery robot.

[0035] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0036] Please refer to Figure 1 , Figure 1 A flowchart of an information prompting method for an interventional surgery robot provided in an embodiment of the present application.

[0037] like Figure 1 As shown, the information prompting method of the interventional surgery robot includes steps S101 to S103.

[0038] Step S101, acquiring a target image to be identified, and identifying object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body, and the medical device includes at least one of a guide wire and a catheter.

[0039] Exemplarily, a camera for acquiring a target image is arranged at a preset position of the multichannel valve body, wherein the preset position is a position that the medical device needs to reach during the process of advancing or retreating. Of course, it is not limited thereto, and the camera device can also be arranged movably relative to the multichannel valve body, so as to observe various parts of the multichannel valve body by moving the position of the camera device. The target object in the target graphic can be the multichannel valve body or the medical device, or the multichannel valve body and the medical device, which are not limited here.

[0040] It can be understood that the multi-channel valve body can be a T-shaped valve, a Y-shaped valve or the like, and is made of a material with a certain degree of transparency, so that the shape of the liquid in the valve body and the position of the medical device can be observed through the multi-channel valve body in the target image.

[0041] Please refer to Figure 2 , Figure 2 The structure diagram of an interventional surgery robot slave terminal provided in an embodiment of the present application. The information prompting method of the interventional surgery robot can be used for Figure 2 The interventional surgery robot shown in the figure is an interventional surgery robot from the end 10, as shown in the figure Figure 2 As shown, the interventional surgical robot is equipped with a multi-channel valve body 101 from the end 10, and the multi-channel valve body 101 is connected to the catheter 102 and the guide wire (not shown). The target image is acquired by a camera device 103 arranged opposite to the multi-channel valve body 101 to identify the state of at least one of the multi-channel valve body 101, the catheter 102 and the guide wire in the target image.

[0042] In some implementations, the step of acquiring a target image to be identified includes:

[0043] Acquiring a real-time image of a preset position of a multi-channel valve body at a slave end of the interventional surgery robot by means of a camera device;

[0044] The real-time image is sent to a processor, and grayscale processing is performed on the real-time image to obtain the target image.

[0045] Exemplarily, in order to ensure that the target image can timely reflect the real-time situation during the operation performed by the interventional robot, a real-time image of a preset position of a multi-channel valve body is acquired at a certain frequency through a camera device; or a video stream consisting of multiple frames of images is acquired, and real-time images are extracted from the video stream at a certain number of frame intervals.

[0046] Exemplarily, the camera device is connected to the processor of the interventional surgery robot slave end, and the real-time image is transmitted to the processor, and the processor pre-processes the real-time image to obtain the target image. Specifically, the pre-processing includes but is not limited to grayscale processing, which converts the real-time image with multiple color channels (such as RBG image) into a grayscale image with only grayscale values, so that the important features in the real-time image are reflected, avoiding the need to process multiple color channels separately in the subsequent image recognition process, and reducing the amount of calculation for identifying the target image.

[0047] Exemplarily, the process of preprocessing the real-time image to obtain the target image may also include image enhancement, image filtering, image transformation (such as resizing, rotation, scaling, cropping, etc.), etc., to reduce the noise in the image, increase the significance of the gradient information in the image, and highlight the multi-channel valve body and instrument in the image, thereby improving the accuracy of subsequent image recognition.

[0048] In some embodiments, a backlight source is disposed at a side of the interventional surgery robot at a preset position of the multi-channel valve body away from the camera device, and the backlight source is spaced apart from the multi-channel valve body by a diffuser.

[0049] like Figure 2 As shown, in order to improve the brightness of the target image, a backlight source 104, such as a light emitting diode, can also be set on the side of the multi-channel valve body 101 away from the camera device 103 to provide background light for the target image and highlight the multi-channel valve body and the instrument in the target image.

[0050] Please refer to Figure 3 , Figure 3 A schematic block diagram of the structure of an interventional surgery robot slave end provided in one embodiment of the present application.

[0051] like Figure 3 As shown, in order to reduce the interference of the background and ambient brightness on the target image, a backlight source 104 and a diffuser 105 are arranged at a preset position where the target image needs to be acquired, and the backlight source 104 provides background brightness for the photographed object, thereby enhancing the clarity of the contour edge features of the photographed object in the target image. The backlight source 104 and the multi-channel valve body are arranged at intervals through the diffuser 105, and the light diffusion effect is enhanced by the roughness of the surface of the diffuser 105, so that the light emitted by the backlight source 104 is more evenly distributed, thereby improving the uniformity of the brightness of the target image, reducing glare and reflection in the target image, and improving the quality of the target image.

[0052] Exemplarily, the color of the diffuser sheet 104 may be white, providing a white background for the target image and improving the image quality of the target image.

[0053] In some embodiments, the step of identifying object features of a target object in the target image includes:

[0054] Determine object edge pixels in the target image based on an edge detection algorithm;

[0055] The shape of the target object in the target image is recognized based on the object edge pixels to obtain a target object contour composed of the object edge pixels.

[0056] Exemplarily, the object features of the target object include the target object contour of the target object, such as the contour of the multi-channel valve body and the contour of the guidewire and / or catheter in the target image, so as to identify the multi-channel valve body and the medical device in the target image according to the target object contour. Specifically, the object edge pixels belonging to the edge of the object in the pixel points of the target image are determined by an edge detection algorithm, and then shape recognition is performed based on the object edge pixels to determine the target object contour to which each object edge pixel belongs, thereby obtaining the target object contour composed of the object edge pixels, so as to subsequently identify the valve body state or the device position state of the target object in the target image according to the target object contour.

[0057] In some implementations, the step of determining object edge pixels in the target image based on an edge detection algorithm includes:

[0058] Calculating the horizontal gradient and vertical gradient of each pixel in the target image based on the grayscale value of each pixel and a preset convolution kernel;

[0059] Determining a target gradient of the pixel point according to the horizontal gradient and the vertical gradient;

[0060] The pixel points whose target gradient is greater than the preset gradient are determined as the object edge pixels.

[0061] Exemplarily, whether a pixel point belongs to the edge of an object in the target image can be determined by calculating the spatial gradient of the brightness of each pixel point. The target gradient includes a horizontal gradient and a vertical gradient, which are used to detect edges in the horizontal direction and the vertical direction, respectively. Specifically, a first convolution kernel Kh is set for the horizontal gradient, and a second convolution kernel Kv is set for the vertical gradient. These two convolution kernels are convolved with each pixel point of the target image, respectively, to obtain the horizontal gradient and vertical gradient of each pixel point. The values ​​of the first convolution kernel Kh and the second convolution kernel Kv are as follows, but are certainly not limited to these and are not limited here.

[0062]

[0063] Exemplarily, the horizontal gradient Gx, the vertical gradient Gy and the target gradient G are calculated by the following formula:

[0064]

[0065] Among them, Gx represents the horizontal gradient of the pixel, Gy represents the vertical gradient of the pixel, G represents the target gradient of the pixel, Kh(x, y) represents the first convolution kernel, Kv(x, y) represents the second convolution kernel, and I(x, y) represents the grayscale value of the pixel (x, y).

[0066] Exemplarily, according to the size of the target gradient, the pixel points in the target image that are object edge pixels can be determined. Specifically, the pixel points with a larger target gradient are determined as object edge pixels. For example, a gradient threshold can be set, and the pixel points with a target gradient greater than the gradient threshold are determined as object edge pixels. The set of object edge pixels is the target object contour.

[0067] In some implementations, the step of performing shape recognition on the target object in the target image based on the object edge pixels to obtain a target object contour composed of the object edge pixels includes:

[0068] For each of the object edge pixels, calculating a shape parameter of at least one shape to which the object edge pixel belongs;

[0069] Determine the number of object edge pixels corresponding to each of the shape parameters;

[0070] According to the number of edge pixels of the object, a shape corresponding to at least one shape parameter is determined as the contour of the target object.

[0071] Exemplarily, the Hough transform can determine the shapes of circles, rectangles, straight lines, etc. composed of object edge pixels. Specifically, the shape to which each object edge pixel belongs is determined respectively. An object edge pixel may belong to multiple shapes (for example, multiple lines pass through the object edge pixel). The number of object edge pixels corresponding to each shape is determined, and the number of object edge pixels corresponding to each shape parameter is calculated in an accumulator. The shape corresponding to the shape parameter with a large number of object edge pixels is considered to be the actual shape in the target image. Among them, the shape is described by the shape parameter, for example, a straight line can be represented by ρ=xcosθ+ysinθ.

[0072] Exemplarily, the object edge pixels can be mapped into the parameter space by the following formula:

[0073] v=x·sinθ+y·cosθ

[0074] h=x·cosθ-y·sinθ

[0075] Among them, v and h represent the coordinates in the accumulator matrix respectively, θ represents radians, and x and y represent the coordinates of the object edge pixels in the target image; v and h can be regarded as components of ρ, and the equation of the line can be determined according to ρ and θ. The equation of the line containing the largest number of object edge pixels can be considered as the equation of the line that actually exists in the target image.

[0076] Exemplarily, after obtaining the target object contour in the target image, since the target image is a grayscale image with pixel values ​​of 0-255, the image corresponding to the target object contour can be normalized, and the pixel value of the target object contour is normalized to the range of 0-1 through the formula normalized_image=image / 255, where image represents the image corresponding to the target object contour, and normalized_image represents the image corresponding to the target object contour after normalization. Through normalization, the amount of calculation for subsequent recognition of the target object contour is reduced, and the convergence speed of the model is improved.

[0077] Step S102: input the object features into a pre-trained neural network model, and output at least the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body.

[0078] Exemplarily, the valve body state and the instrument position state are identified based on the neural network model. Specifically, the target image can be input into a single neural network model of multi-task learning (MTL), or the neural network model can be trained separately for different task scenarios, which is not limited here.

[0079] In some embodiments, the step of inputting the object features into a pre-trained neural network model to obtain the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body includes:

[0080] Inputting the object features into a pre-trained first neural network sub-model, and outputting the first neural network sub-model to determine the valve body state in the target image; and / or,

[0081] The object features are input into a pre-trained second neural network sub-model, and the second neural network sub-model is output to determine the position state of the instrument in the target image.

[0082] Exemplarily, after the processor (CPU) completes the preprocessing of the target image, the object features in the target image, such as the target object contour, are input into the neural network, wherein the neural network model can be set in a neural network processor (Neural Network Processing Unit, NPU). NPU has high parallelism, low power consumption, specific optimization and flexible configuration, and can accelerate the calculation of neural network tasks.

[0083] Exemplarily, in order to reduce the complexity of the neural network model and avoid interference between different tasks, the information prompting method for the interventional surgical robot provided in the embodiment of the present application trains the first neural network sub-model and the second neural network sub-model for valve body state recognition and instrument position state recognition, respectively.

[0084] Exemplarily, the first neural network sub-model and the second neural network sub-model can be trained separately using sample images with different valve body states and instrument position states until model parameters of the first neural network sub-model and the second neural network sub-model meet preset conditions.

[0085] Specifically, the first neural network sub-model is used to identify the liquid in the multi-channel valve body and determine whether there is blood return in the valve body. Under normal circumstances, a liquid infusion channel is provided in the multi-channel valve body, and the liquid in the liquid infusion channel is a colorless and transparent liquid, such as contrast agent, heparin water, normal saline, etc.; because the liquid in the liquid infusion channel of the multi-channel valve body is connected to the human blood vessels through the liquid circuit during the interventional surgery, when the liquid circuit drip pressure is abnormal, the human blood may flow back into the valve body, causing blood loss in the human body. At this time, the liquid in the multi-channel valve body presents different color forms due to the presence of blood, such as red, light brown, etc. Therefore, the valve body state in the target image is identified by the first neural network sub-model to confirm whether blood return occurs in the valve body.

[0086] The second neural network sub-model is used to identify the medical device in the multi-channel valve body and determine the position of the medical device in the multi-channel valve body, wherein the medical device can be at least one of a guidewire and a catheter. Taking the guidewire as an example, before injecting the contrast agent into the multi-channel valve body, the guidewire needs to be withdrawn to prevent the guidewire from affecting the injection of the contrast agent. However, if the guidewire is completely withdrawn from the multi-channel valve body, it may be necessary to re-insert the guidewire into the multi-channel valve body later, resulting in cumbersome operation; in order to avoid repeated insertion of the guidewire, the guidewire is withdrawn to a preset position of the multi-channel valve body before injecting the contrast agent, and the preset position is a position where no contrast agent flows through the valve body; in this scenario, the second neural network sub-model can be used to identify whether the guidewire in the target image is withdrawn to the preset position. Of course, it is not limited to this. The medical device can also be a catheter, or a guidewire and a catheter. For example, when replacing the catheter, the catheter needs to be withdrawn through the multi-channel valve body. At this time, the second neural network sub-model can be used to identify whether the catheter in the target image is withdrawn to the preset position.

[0087] like Figure 3 As shown, the target image acquired by the camera device 103 is pre-processed by grayscale and the like, and then input into the neural network model in the NPU for recognition. The recognition result and the recognized target image can be stored in the memory so as to record the surgical process in real time. The memory is connected to the CPU via a bus, and the CPU sends information prompts to the main end of the interventional surgery robot through an external interface according to the recognized valve body state and instrument position state.

[0088] Step S103: if it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body, a first information prompting operation is executed; if it is determined according to the valve body state that blood reflux does not occur in the multi-channel valve body, the first information prompting operation is not executed; and / or, if it is determined according to the device position state that the medical device has retreated to the target position, a second information prompting operation is executed; if it is determined according to the device position state that the medical device has not retreated to the target position, the second information prompting operation is not executed.

[0089] For example, in the process of performing interventional surgery by using an interventional surgical robot instead of manual labor, such as performing compartment surgery or remote surgery, it is usually necessary to set up staff to monitor the surgery in real time at the slave end so that they can deal with emergencies in a timely manner. The interventional surgical robot control method provided in the embodiment of the present application sends corresponding information prompts for the valve body status and the instrument position status, so that the status of the surgery can be provided to the operator in real time, thereby reducing the labor cost in compartment surgery and improving the intelligence of interventional surgery.

[0090] Please refer to Figure 4-5. Figure 4a , Figure 4bA schematic diagram of a valve body state in a target image provided by an embodiment of the present application; Figure 5a , Figure 5b A schematic diagram of the position status of an instrument in a target image provided in one embodiment of the present application. Figure 4a The valve body status shown in the figure is a state where there is no blood return. Figure 4b The valve body status shown in the figure is the state of blood return. Figure 5a The device position status shown in the figure is the state where the guidewire has not been withdrawn to the target position. Figure 5b The device position status shown in the figure is the status when the guidewire is withdrawn to the target position.

[0091] For example, if it is determined based on the valve body state that there is blood reflux in the multi-channel valve body, in order to prevent the human body from continuing to lose blood, the first information prompt related operation is executed, or when it is determined that there is blood reflux in the multi-channel valve body, the robot is automatically controlled to stop urgently to avoid affecting the safety of the interventional room operation. Among them, the first information prompt related operation is to output a prompt message to the operator to prompt the operator that there is blood reflux, such as text information or sound and light prompt information in a dialog box, to promptly notify the operator of the blood reflux phenomenon during the operation so that corresponding measures can be taken. In this way, there is no need to set up personnel in the operating room to monitor the blood reflux phenomenon.

[0092] For example, when the guidewire is withdrawn to the target position according to the target image, it indicates that the contrast agent injection is ready, and the second information prompt related operation is performed, or when the guidewire is withdrawn to the target position according to the target image, the robot is automatically controlled to stop suddenly to prevent the doctor from not paying attention, the guidewire is withdrawn all the time, completely withdrawn from the T valve, and falling to the ground to cause contamination, and the guidewire needs to be re-threaded, which greatly prolongs the operation time. Among them, the second information prompt related operation is to output a prompt information for prompting the operator to withdraw the guidewire to the target position. After receiving the second information prompt, the processor can control the interventional surgery robot to perform the contrast agent injection operation.

[0093] Please refer to Figure 6 , Figure 6 A schematic diagram of a process for training a state recognition model provided in an embodiment of the present application.

[0094] like Figure 6As shown, a target model for implementing the information prompting method of the interventional surgical robot provided in the embodiment of the present application is trained by pre-set sample images and sample states corresponding to the sample images. Among them, the sample images are pre-processed, including image filtering, graying, enhancement processing and other operations; the pre-processed images are subjected to object feature extraction to obtain the contour of the target object in the image, and the contour of the target object is normalized; after the normalization, the contour of the target object is subjected to state recognition to obtain the target state Xm, and the target state is compared with the sample state Xn of the sample image. If Xm matches Xn, it indicates that the model can be used for state recognition of the target image, otherwise the model continues to be trained. Among them, the comparison between Xm and Xn can be to calculate the accuracy of Xm based on Xn. When the accuracy is greater than the preset accuracy, it is determined that Xm matches Xn. Of course, it is not limited to this and is not limited here.

[0095] The information prompting method of the interventional surgical robot provided in the embodiment of the present application obtains the target image to be identified and identifies the object features of the target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body, and the medical device includes at least one of a guide wire and a catheter; the object features are input into a preset neural network to obtain the valve body state of the multi-channel valve body and the device position state of the medical device in the multi-channel valve body; and the information prompting operation is performed according to the valve body state and / or the device position state. Since the target image including the multi-channel valve body and the medical device is acquired in real time, the valve body state and the device position state in the target image are identified, the blood reflux in the valve body is detected according to the valve body state, and the position of the device in the multi-channel valve body is determined according to the device position state, thereby performing the corresponding information prompting operation, reducing the manpower requirements during the interventional surgery and improving the safety and intelligence of the interventional surgical robot.

[0096] The present application also provides an information prompting device for an interventional surgery robot, which is used to implement the method described in any of the above embodiments. Figure 7 As shown, including:

[0097] An image recognition module 200, configured to acquire a target image to be recognized and recognize object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body, wherein the medical device includes at least one of a guide wire and a catheter;

[0098] A state information acquisition module 300, for inputting the object features into a pre-trained neural network model, and outputting at least the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body;

[0099] The information prompt module 400 is used to execute a first information prompt related operation when it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body; not execute the first information prompt related operation when it is determined according to the valve body state that blood reflux does not occur in the multi-channel valve body; and / or execute a second information prompt related operation when it is determined according to the device position state that the medical device has retreated to the target position; and not execute the second information prompt related operation when it is determined according to the device position state that the medical device has not retreated to the target position.

[0100] As described above, it can be understood that the various components of the information prompting device of the interventional surgery robot proposed in the present application can realize the functions of any one of the above-mentioned information prompting methods of the interventional surgery robot.

[0101] The embodiment of the present application also provides an interventional surgery robot, comprising: an interventional surgery robot slave end, an interventional surgery robot master end and a controller;

[0102] The interventional surgical robot slave end is provided with a valve body assembly mounting portion, the valve body assembly mounting portion is installed with a multi-channel valve body, the multi-channel valve body is used to accommodate at least one of a catheter and a guide wire, and a camera device is also provided at a position opposite to the valve body assembly mounting portion and the multi-channel valve body, for acquiring a target image of the multi-channel valve body;

[0103] The interventional surgery robot master end is used to receive the user's operation instructions and control the interventional surgery robot slave end through the controller;

[0104] The controller is used to execute the information prompting method of the interventional surgery robot.

[0105] Exemplarily, a backlight source is disposed at a slave end of the interventional surgery robot on a side of the multi-channel valve body away from the imaging device, and the backlight source and the multi-channel valve body are spaced apart by a diffuser.

[0106] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the controller of the interventional surgery robot described above can refer to the corresponding process in the aforementioned information prompt method embodiment of the interventional surgery robot, and will not be repeated here.

[0107] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the information prompt method of the interventional surgical robot of the present application.

[0108] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0109] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0110] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0111] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An information prompting method for an interventional surgery robot, characterized in that: The method comprises: Acquire a target image to be identified, and identify object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body; Inputting the object features into a pre-trained neural network model to output at least a valve body state of the multi-channel valve body and / or an instrument position state of the medical instrument in the multi-channel valve body; If it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body, executing a first information prompt related operation; If it is determined according to the valve body state that no blood reflux occurs in the multi-channel valve body, then the first information prompt related operation is not performed; and / or, If it is determined according to the device position state that the medical device has retreated to the target position, executing a second information prompt related operation; If it is determined according to the device position status that the medical device has not retreated to the target position, the second information prompt related operation is not performed.

2. The information prompting method of the interventional surgery robot according to claim 1, characterized in that: The step of identifying object features of a target object in the target image comprises: Determine object edge pixels in the target image based on an edge detection algorithm; The shape of the target object in the target image is recognized based on the object edge pixels to obtain a target object contour composed of the object edge pixels.

3. The information prompting method of the interventional surgery robot according to claim 2, characterized in that: The step of determining object edge pixels in the target image based on an edge detection algorithm comprises: Calculating the horizontal gradient and vertical gradient of each pixel in the target image based on the grayscale value of each pixel and a preset convolution kernel; Determining a target gradient of the pixel point according to the horizontal gradient and the vertical gradient; The pixel points whose target gradient is greater than the preset gradient are determined as the object edge pixels.

4. The information prompting method of the interventional surgery robot according to claim 2, characterized in that: The step of performing shape recognition on the target object in the target image based on the object edge pixels to obtain the target object contour composed of the object edge pixels comprises: For each of the object edge pixels, calculating a shape parameter of at least one shape to which the object edge pixel belongs; Determine the number of object edge pixels corresponding to each of the shape parameters; According to the number of edge pixels of the object, a shape corresponding to at least one shape parameter is determined as the contour of the target object.

5. The information prompting method of the interventional surgery robot according to claim 1, characterized in that: The step of inputting the object features into a pre-trained neural network model to at least output the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body comprises: Inputting the object features into a pre-trained first neural network sub-model to at least output the valve body state in the target image; and / or, The object features are input into a pre-trained second neural network sub-model to at least output the instrument position status in the target image.

6. The information prompting method of the interventional surgery robot according to claim 1, characterized in that: The step of acquiring the target image to be identified comprises: Acquiring a real-time image of a preset position of a multi-channel valve body at a slave end of the interventional surgery robot; The real-time image is sent to a processor, and grayscale processing is performed on the real-time image to obtain the target image.

7. An information prompting device for an interventional surgery robot, characterized in that: The information prompting device comprises: An image recognition module, used to acquire a target image to be recognized and recognize object features of a target object in the target image, wherein the target object at least includes a multi-channel valve body and / or a medical device located in the multi-channel valve body; A state information acquisition module, used for inputting the object features into a pre-trained neural network model, and outputting at least the valve body state of the multi-channel valve body and / or the device position state of the medical device in the multi-channel valve body; An information prompt module is used to execute a first information prompt-related operation when it is determined according to the valve body state that blood reflux occurs in the multi-channel valve body; not execute the first information prompt-related operation when it is determined according to the valve body state that blood reflux does not occur in the multi-channel valve body; and / or execute a second information prompt-related operation when it is determined according to the device position state that the medical device has retreated to the target position; and not execute the second information prompt-related operation when it is determined according to the device position state that the medical device has not retreated to the target position.

8. An interventional surgery robot, characterized in that: include: Interventional surgery robot slave terminal, interventional surgery robot master terminal and controller; The interventional surgical robot slave end is provided with a valve body assembly mounting portion, the valve body assembly mounting portion is installed with a multi-channel valve body, the multi-channel valve body is used to accommodate medical equipment, and a camera device is also provided at a position opposite to the valve body assembly mounting portion and the multi-channel valve body, for acquiring a target image of the multi-channel valve body; The interventional surgery robot master end is used to receive the user's operation instructions and control the interventional surgery robot slave end through the controller; The controller is used to execute the information prompting method of the interventional surgery robot as described in any one of claims 1-6.

9. The interventional surgery robot according to claim 8, characterized in that: The interventional surgery robot slave end is provided with a backlight source on a side of the multi-channel valve body away from the camera device, and the backlight source and the multi-channel valve body are spaced apart by a diffuser.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the information prompting method for an interventional surgical robot according to any one of claims 1 to 6 are implemented.

Images