A device and method for automatically flushing an endoscope for an endoscope holding robot

Abstract

The present invention relates to the technical field of automatic endoscope flushing, and more specifically, to a device and method for automatic endoscope flushing of an endoscope holding robot. For an endoscope that is blurred during surgery, the endoscope is automatically cleaned by holding the robot arm. During the movement, a series of target nodes are set and obstacle avoidance paths are planned between adjacent target nodes, so that the robot arm automatically holds the endoscope and moves to automatically clean it. Through the blur judgment of the endoscope, the endoscope can be cleaned without human intervention.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic endoscope flushing, and more specifically, to a device and method for automatically flushing an endoscope used in an endoscope holding robot. Background Art

[0002] The clearing process of an endoscope is not a simple movement from one point to another, but a more complicated process: the endoscope needs to be withdrawn from the cavity where the operation is performed, and then moved to the sink for cleaning. After cleaning, it is necessary to move to the wiping cloth to wipe the endoscope lens dry, and then it is necessary to determine whether the lens is clean. If it is not clean, it needs to be cleaned and dried again. After cleaning the lens, the endoscope needs to be reinserted into the surgical cavity. The scenario of endoscope cleaning is that the lens is contaminated by blood, body fluids, etc., causing blurred images. Before the endoscope is cleaned, it is impossible to obtain a clear endoscopic image as a basis for control. The current endoscope holding robots cannot solve the problem of automatic flushing of endoscopes and achieve the clarity of endoscopes. They all need to be flushed under manual operation. Summary of the invention

[0003] In order to overcome the defects in the above-mentioned prior art, the present invention provides a method for automatically flushing an endoscope for an endoscope holding robot, which can perform path planning and movement when the endoscope image is blurred, thereby achieving endoscope cleaning.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for automatically flushing an endoscope for an endoscope holding robot, comprising the following steps:

[0005] S1. Before the operation begins, target nodes are set inside the surgical cavity, outside the cavity, at the cleaning basin, at the wipes, and above the wipes;

[0006] S2. Set up a depth camera in the operating room to observe the endoscope, the robotic arm, the target node and other obstacles and find their target positions respectively;

[0007] S3. During the operation, when the endoscope lens is contaminated and the image is blurred, the operator presses the trigger switch for automatic cleaning;

[0008] S4. The industrial computer receives the trigger information, records the current position as the node in the cavity, and controls the robotic arm to hold the endoscope backward and exit from the cavity;

[0009] S5. The depth camera observes the endoscope lens. When the endoscope lens is identified, it indicates that the endoscope has completely withdrawn from the cavity and reached the node outside the cavity;

[0010] S6. Set the node outside the cavity as the starting point and the cleaning basin as the target point. According to the node position information returned by the depth camera and the position information of the obstacles, the industrial computer plans the movement path of the endoscope, and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the cleaning basin and wash the endoscope with water.

[0011] S7. Set the cleaning basin as the starting point and the wiping cloth as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the wiping cloth and wipe off the water on the lens.

[0012] S8. Set the wiping cloth as the starting point and the top of the wiping cloth as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the top of the wiping cloth.

[0013] S9. The endoscope image is blurred above the wiping cloth. If the image is clear, it means that it is cleaned and the next step is entered; if the image is blurred, it means that it is not cleaned, and the endoscope is moved to the water basin again for cleaning, and then steps S7 to S9 are repeated;

[0014] S10. Set the top of the wiping cloth as the starting point and the outside of the cavity as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope and reach the outside of the cavity.

[0015] S11. Control the endoscope to re-enter the node in the cavity, use the endoscopic image to control the insertion position and angle of the lens, and continue to complete the operation.

[0016] In the present invention, a system consisting of five main devices including an endoscope, a depth camera, a robotic arm, an industrial computer, and a cleaning component (cleaning basin, wiping cloth) is constructed. Target nodes are set inside the surgical cavity (D), outside the cavity (C), at the cleaning basin (B), at the wiping cloth (A), and above the wiping cloth (E). A depth camera is set up in the operating room to observe the endoscope, the robotic arm, several target nodes and other obstacles, and construct their position coordinates in three-dimensional space. The industrial computer collects information from the endoscope and the depth camera, and controls the robotic arm according to the path planning results. The endoscope is held by the robotic arm, and the endoscope is moved to several target nodes in turn to clean the endoscope. Before moving between two nodes, the industrial computer plans the shortest path to avoid obstacles based on the obstacle information between the nodes, and then controls the robotic arm to move. At the same time, the industrial computer decides whether to clean again based on whether the endoscopic image is blurred. After cleaning, the endoscope is re-inserted into the surgical cavity, allowing the doctor to continue the operation.

[0017] Furthermore, when the depth camera obtains the position of an object, the following steps are specifically included:

[0018] The depth camera obtains the depth information in the picture and transmits the picture and depth information back to the industrial computer;

[0019] The industrial computer uses a semantic segmentation algorithm to segment the objects in the image sent back by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object;

[0020] Using the depth camera position as a reference, the coordinates of each object in the spatial coordinate system are inferred based on the depth information of each object.

[0021] Furthermore, in step S5, the depth camera identifies whether the endoscope is completely withdrawn from the cavity, which specifically includes the following steps:

[0022] The neural network is trained using photos with the endoscope lens fully visible and photos with the lens not fully visible as binary classification training samples;

[0023] When the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image;

[0024] When the complete endoscope lens is detected, a signal is sent that the endoscope has been completely withdrawn from the cavity, allowing the robotic arm to perform the next stage of movement.

[0025] Furthermore, the industrial computer plans the motion path of the robot arm, specifically including the following steps:

[0026] A solution to obtain the position of objects using a depth camera to obtain the position of the robotic arm, endoscope, and each object in the scene;

[0027] The process of cleaning the endoscope is divided into several small processes according to the set nodes; for each small process, the starting point and end point of the endoscope movement are input;

[0028] For each small process, the industrial computer plans the shortest path for the robot arm to avoid obstacles based on the information of the starting point, the end point and the obstacles in the middle, according to the existing path planning algorithm;

[0029] The industrial computer solves the path into the motion trajectory of the robot arm and sends instructions to control the robot arm to move;

[0030] During the movement, the depth camera collects the location information of the object in real time, and the industrial computer determines the possible collision based on the returned information and then corrects the path.

[0031] Furthermore, the fuzzy determination of the endoscope image specifically includes the following steps:

[0032] The industrial computer uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscope image, and then calculates the variance to find the fuzzy score of the image;

[0033] Determine whether the blur score of the image exceeds a threshold, if so, determine it as a blurry image;

[0034] When the endoscope images for three consecutive seconds are judged to be blurred images, it is determined that the current endoscope needs to be cleaned.

[0035] The present invention also provides a device for automatically flushing an endoscope for an endoscope holding robot, comprising:

[0036] Target node setting module: used to set target nodes inside the surgical cavity, outside the cavity, at the cleaning basin, at the wiping cloth, and above the wiping cloth before the operation begins;

[0037] Depth camera: used to observe the endoscope, robotic arm, target node and other obstacles and find their target positions respectively;

[0038] Robotic arm: used to hold the endoscope and move it, and connect and communicate with the industrial computer;

[0039] Endoscope: used to collect images inside the cavity and transmit them to the industrial computer;

[0040] Path planning module: used to plan the movement path of the robot arm according to the images collected by the depth camera and endoscope and the position of the set target node;

[0041] Industrial computer: used to control the robot arm to hold the endoscope and move it according to the moving path planned by the path planning module, so that the endoscope can be withdrawn from the cavity to the outside of the cavity, and then moved to the cleaning basin for cleaning and the wiping cloth for drying. After the blur judgment module determines that the endoscope lens is clear, the endoscope is moved into the cavity;

[0042] Blur judgment module: used to judge whether the endoscope lens is clear.

[0043] Furthermore, it also includes an object position acquisition unit, which is used to obtain depth information in the picture according to the depth camera, and transmit the picture and the depth information back to the industrial computer. The industrial computer uses a semantic segmentation algorithm to segment the objects in the picture transmitted by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object, using the depth camera position as a reference, and infers the coordinates of each object in the spatial coordinate system according to the depth information of each object.

[0044] Furthermore, it also includes an endoscope recognition module, which is used to use photos in which the endoscope lens appears completely and photos in which the lens does not appear completely as binary classification training samples to train the neural network; when the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image; when a complete endoscope lens is detected, a signal is sent that the endoscope has completely exited the cavity, so that the robotic arm can perform the next stage of movement.

[0045] Furthermore, the path planning module includes:

[0046] Cleaning path planning unit: used to set the node outside the cavity as the starting point and the cleaning basin as the target point, plan the endoscope movement path according to the node position information returned by the depth camera and the position information of the obstacles, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer, control the robotic arm to hold the endoscope to the cleaning basin, and wash the endoscope with water;

[0047] Wiping path planning unit: used to set the wiping cloth as the starting point and the top of the wiping cloth as the target point, plan the endoscope moving path, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the top of the wiping cloth;

[0048] Return path planning unit: used to set the top of the wiping cloth as the starting point and the outside of the cavity as the target point, plan the movement path of the endoscope, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the outside of the cavity;

[0049] Obstacle avoidance path planning module: a solution for using the depth camera to obtain the position of objects, the position of the robotic arm, the endoscope and objects in the scene, and the process of cleaning the endoscope is divided into several small processes according to the nodes therein. For each small process, the starting point and end point of the endoscope movement are input; for each small process, according to the starting point and end point and the obstacle information in the middle, according to the existing path planning algorithm, the shortest path for the robotic arm to avoid obstacles is planned; the path is solved into the motion trajectory of the robotic arm, and instructions are sent to control the movement of the robotic arm; during the movement, the possible collision is judged based on the object position information collected in real time by the depth camera, and the path is corrected.

[0050] Furthermore, the fuzzy judgment module uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscopic image respectively, and then calculates the variance to obtain the fuzzy score of the image; determines whether the fuzzy score of the image exceeds a threshold, and if so, determines it to be a fuzzy image; when the endoscopic images for 3 consecutive seconds are determined to be fuzzy images, it is determined that the current endoscope needs to be cleaned.

[0051] Compared with the prior art, the beneficial effects are: the present invention provides a device and method for automatically flushing an endoscope for an endoscope holding robot, which can automatically clean an endoscope that is blurred during surgery by holding it with a robotic arm. During the movement, a series of target nodes are set and obstacle avoidance paths are planned between adjacent target nodes, thereby enabling the robotic arm to automatically hold the endoscope and move it for automatic cleaning; through blur judgment of the endoscope, the endoscope can be cleaned without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The present invention is described in one of the embodiments in combination with the specific implementation methods. Among them, the drawings are only used for exemplary descriptions, and only schematic diagrams are shown, not physical drawings, and cannot be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0054] In the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right" and the like indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only used for exemplary explanations and cannot be understood as a limitation of this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances. In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text is to include three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes that satisfy both A and B.

[0055] Embodiment 1:

[0056] like Figure 1As shown, a method for automatically flushing an endoscope for an endoscope holding robot comprises the following steps:

[0057] S1. Before the operation begins, target nodes are set inside the surgical cavity, outside the cavity, at the cleaning basin, at the wipes, and above the wipes;

[0058] S2. Set up a depth camera in the operating room to observe the endoscope, the robotic arm, the target node and other obstacles and find their target positions respectively;

[0059] S3. During the operation, when the endoscope lens is contaminated and the image is blurred, the operator presses the trigger switch for automatic cleaning;

[0060] S4. The industrial computer receives the trigger information, records the current position as the node in the cavity, and controls the robotic arm to hold the endoscope backward and exit from the cavity;

[0061] S5. The depth camera observes the endoscope lens. When the endoscope lens is identified, it indicates that the endoscope has completely withdrawn from the cavity and reached the node outside the cavity;

[0062] S6. Set the node outside the cavity as the starting point and the cleaning basin as the target point. According to the node position information returned by the depth camera and the position information of the obstacles, the industrial computer plans the movement path of the endoscope, and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the cleaning basin and wash the endoscope with water.

[0063] S7. Set the cleaning basin as the starting point and the wiping cloth as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the wiping cloth and wipe off the water on the lens.

[0064] S8. Set the wiping cloth as the starting point and the top of the wiping cloth as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the top of the wiping cloth.

[0065] S9. The endoscope image is blurred above the wiping cloth. If the image is clear, it means that it is cleaned and the next step is entered; if the image is blurred, it means that it is not cleaned, and the endoscope is moved to the water basin again for cleaning, and then steps S7 to S9 are repeated;

[0066] S10. Set the top of the wiping cloth as the starting point and the outside of the cavity as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope and reach the outside of the cavity.

[0067] S11. Control the endoscope to re-enter the node in the cavity, use the endoscopic image to control the insertion position and angle of the lens, and continue to complete the operation.

[0068] In the present invention, a system consisting of five main devices including an endoscope, a depth camera, a robotic arm, an industrial computer, and a cleaning component (cleaning basin, wiping cloth) is constructed. Target nodes are set inside the surgical cavity (D), outside the cavity (C), at the cleaning basin (B), at the wiping cloth (A), and above the wiping cloth (E). A depth camera is set up in the operating room to observe the endoscope, the robotic arm, several target nodes and other obstacles, and construct their position coordinates in three-dimensional space. The industrial computer collects information from the endoscope and the depth camera, and controls the robotic arm according to the path planning results. The endoscope is held by the robotic arm, and the endoscope is moved to several target nodes in turn to clean the endoscope. Before moving between two nodes, the industrial computer plans the shortest path to avoid obstacles based on the obstacle information between the nodes, and then controls the robotic arm to move. At the same time, the industrial computer decides whether to clean again based on whether the endoscopic image is blurred. After cleaning, the endoscope is re-inserted into the surgical cavity, allowing the doctor to continue the operation.

[0069] Specifically, when the depth camera obtains the position of an object, the following steps are included:

[0070] The depth camera obtains the depth information in the picture and transmits the picture and depth information back to the industrial computer;

[0071] The industrial computer uses a semantic segmentation algorithm to segment the objects in the image sent back by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object;

[0072] Using the depth camera position as a reference, the coordinates of each object in the spatial coordinate system are inferred based on the depth information of each object.

[0073] Wherein, in the step S5, the depth camera identifies whether the endoscope is completely withdrawn from the cavity, including the following steps:

[0074] The neural network is trained using photos with the endoscope lens fully visible and photos with the lens not fully visible as binary classification training samples;

[0075] When the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image;

[0076] When the complete endoscope lens is detected, a signal is sent that the endoscope has been completely withdrawn from the cavity, allowing the robotic arm to perform the next stage of movement.

[0077] In addition, the industrial computer plans the motion path of the robot arm, which specifically includes the following steps:

[0078] A solution to obtain the position of objects using a depth camera to obtain the position of the robotic arm, endoscope, and each object in the scene;

[0079] The process of cleaning the endoscope is divided into several small processes according to the set nodes; for each small process, the starting point and end point of the endoscope movement are input;

[0080] For each small process, the industrial computer plans the shortest path for the robot arm to avoid obstacles based on the information of the starting point, the end point and the obstacles in the middle, according to the existing path planning algorithm;

[0081] The industrial computer solves the path into the motion trajectory of the robot arm and sends instructions to control the robot arm to move;

[0082] During the movement, the depth camera collects the location information of the object in real time, and the industrial computer determines the possible collision based on the returned information and then corrects the path.

[0083] The fuzzy judgment of the endoscope image specifically includes the following steps:

[0084] The industrial computer uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscope image, and then calculates the variance to find the fuzzy score of the image;

[0085] Determine whether the blur score of the image exceeds a threshold, if so, determine it as a blurry image;

[0086] When the endoscope images for three consecutive seconds are judged to be blurred images, it is determined that the current endoscope needs to be cleaned.

[0087] Example 2

[0088] This embodiment provides a device for automatically flushing an endoscope for an endoscope holding robot, comprising:

[0089] Target node setting module: used to set target nodes inside the surgical cavity, outside the cavity, at the cleaning basin, at the wiping cloth, and above the wiping cloth before the operation begins;

[0090] Depth camera: used to observe the endoscope, robotic arm, target node and other obstacles and find their target positions respectively;

[0091] Robotic arm: used to hold the endoscope and move it, and connect and communicate with the industrial computer;

[0092] Endoscope: used to collect images inside the cavity and transmit them to the industrial computer;

[0093] Path planning module: used to plan the movement path of the robot arm according to the images collected by the depth camera and endoscope and the position of the set target node;

[0094] Industrial computer: used to control the robot arm to hold the endoscope and move it according to the moving path planned by the path planning module, so that the endoscope can be withdrawn from the cavity to the outside of the cavity, and then moved to the cleaning basin for cleaning and the wiping cloth for drying. After the blur judgment module determines that the endoscope lens is clear, the endoscope is moved into the cavity;

[0095] Blur judgment module: used to judge whether the endoscope lens is clear.

[0096] Specifically, it also includes an object position acquisition unit, which is used to obtain depth information in the picture according to the depth camera, and transmit the picture and the depth information back to the industrial computer. The industrial computer uses a semantic segmentation algorithm to segment the objects in the picture sent back by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object. The depth camera position is used as a reference to infer the coordinates of each object in the spatial coordinate system according to the depth information of each object.

[0097] Wherein, the path planning module includes:

[0098] Cleaning path planning unit: used to set the node outside the cavity as the starting point and the cleaning basin as the target point, plan the endoscope movement path according to the node position information returned by the depth camera and the position information of the obstacles, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer, control the robotic arm to hold the endoscope to the cleaning basin, and wash the endoscope with water;

[0099] Wiping path planning unit: used to set the wiping cloth as the starting point and the top of the wiping cloth as the target point, plan the endoscope moving path, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the top of the wiping cloth;

[0100] Return path planning unit: used to set the top of the wiping cloth as the starting point and the outside of the cavity as the target point, plan the movement path of the endoscope, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the outside of the cavity;

[0101] Obstacle avoidance path planning module: a solution for using the depth camera to obtain the position of objects, the position of the robotic arm, the endoscope and objects in the scene, and the process of cleaning the endoscope is divided into several small processes according to the nodes therein. For each small process, the starting point and end point of the endoscope movement are input; for each small process, according to the starting point and end point and the obstacle information in the middle, according to the existing path planning algorithm, the shortest path for the robotic arm to avoid obstacles is planned; the path is solved into the motion trajectory of the robotic arm, and instructions are sent to control the movement of the robotic arm; during the movement, the possible collision is judged based on the object position information collected in real time by the depth camera, and the path is corrected.

[0102] In addition, the fuzzy judgment module uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscopic image respectively, and then calculates the variance to obtain the fuzzy score of the image; determines whether the fuzzy score of the image exceeds a threshold, and if so, determines it to be a fuzzy image; when the endoscopic images for 3 consecutive seconds are all determined to be fuzzy images, it is determined that the current endoscope needs to be cleaned.

[0103] Example 3

[0104] The present embodiment is different from the other parts of the housing of Embodiment 2 in that the present embodiment also includes an endoscope recognition module, which is used to train the neural network by using photos in which the endoscope lens appears completely and photos in which the lens does not appear completely as binary classification training samples; when the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image; when a complete endoscope lens is detected, a signal is sent that the endoscope has completely exited the cavity, so that the robotic arm can perform the next stage of movement.

[0105] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for automatically flushing an endoscope for an endoscope holding robot, characterized in that: The following steps are involved: S1. Before the operation begins, target nodes are set inside the surgical cavity, outside the cavity, at the cleaning basin, at the wipes, and above the wipes; S2. Set up a depth camera in the operating room to observe the endoscope, robotic arm, target node and other obstacles and find their target positions respectively; S3. During surgery, when the endoscope lens is contaminated and the image is blurred, the operator presses the trigger switch for automatic cleaning; S4. The industrial computer receives the trigger information, records the current position as the node in the cavity, and controls the robotic arm to hold the endoscope backward and exit from the cavity; S5. The depth camera observes the endoscope lens. When the endoscope lens is identified, it indicates that the endoscope has completely withdrawn from the cavity and reached the node outside the cavity; S6. Set the node outside the cavity as the starting point and the cleaning basin as the target point. According to the node position information returned by the depth camera and the position information of the obstacles, the industrial computer plans the movement path of the endoscope, and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the cleaning basin and wash the endoscope with water. S7. Set the cleaning basin as the starting point and the wiping cloth as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the wiping cloth and wipe the water off the lens. S8. The wipe cloth is set as the starting point and the top of the wipe cloth is set as the target point. The industrial computer plans the movement path of the endoscope and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope to the top of the wipe cloth. S9. The endoscope image is blurred above the wiping cloth. If the image is clear, it means that it is clean and proceed to the next step; if the image is blurred, it means that it is not clean, then the endoscope is moved to the water basin again for cleaning, and then steps S7 to S9 are repeated; S10. The starting point is set above the wiping cloth, and the target point is set outside the cavity. The industrial computer plans the movement path of the endoscope, and then converts it into the movement path of the robotic arm. Then, the industrial computer issues a command to control the robotic arm to hold the endoscope and reach outside the cavity. S11. Control the endoscope to re-enter the node in the cavity, use the endoscope image to control the insertion position and angle of the lens, and continue to complete the operation; Among them, the industrial computer plans the motion path of the robot arm, which specifically includes the following steps: A solution to obtain the position of objects using a depth camera to obtain the position of the robotic arm, endoscope, and each object in the scene; The process of cleaning the endoscope is divided into several small processes according to the set nodes; for each small process, the starting point and end point of the endoscope movement are input; For each small process, the industrial computer plans the shortest path for the robot arm to avoid obstacles based on the information of the starting point, the end point and the obstacles in the middle, according to the existing path planning algorithm; The industrial computer solves the path into the motion trajectory of the robot arm and sends instructions to control the robot arm to move; During the movement, the depth camera collects the position information of the object in real time, and the industrial computer determines the possible collision based on the returned information and then corrects the path; The fuzzy judgment of the endoscope image specifically includes the following steps: The industrial computer uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscope image, and then calculates the variance to find the fuzzy score of the image; Determine whether the blur score of the image exceeds a threshold, if so, determine it as a blurry image; When the endoscope images for three consecutive seconds are judged to be blurred images, it is determined that the current endoscope needs to be cleaned.

2. The method for automatically flushing an endoscope for an endoscope holding robot according to claim 1, characterized in that: When the depth camera obtains the position of an object, the following steps are specifically included: The depth camera obtains the depth information in the picture and transmits the picture and depth information back to the industrial computer; The industrial computer uses a semantic segmentation algorithm to segment the objects in the image sent back by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object; Using the depth camera position as a reference, the coordinates of each object in the spatial coordinate system are inferred based on the depth information of each object.

3. The method for automatically flushing an endoscope for an endoscope holding robot according to claim 1, characterized in that: In step S5, the depth camera identifies whether the endoscope is completely withdrawn from the cavity, which specifically includes the following steps: The neural network is trained using photos with the endoscope lens fully visible and photos with the lens not fully visible as binary classification training samples; When the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image; When the complete endoscope lens is detected, a signal is sent that the endoscope has been completely withdrawn from the cavity, allowing the robotic arm to perform the next stage of movement.

4. A device for automatically flushing an endoscope for an endoscope holding robot, characterized in that: include: Target node setting module: used to set target nodes inside the surgical cavity, outside the cavity, at the cleaning basin, at the wiping cloth, and above the wiping cloth before the operation begins; Depth camera: used to observe the endoscope, robotic arm, target node and other obstacles and find their target positions respectively; Robotic arm: used to hold the endoscope and move it, and connect with the industrial computer and communicate with each other; Endoscope: used to collect images inside the cavity and transmit them to the industrial computer; Path planning module: used to plan the movement path of the robot arm according to the images collected by the depth camera and endoscope and the position of the set target node; Industrial computer: used to control the robot arm to hold the endoscope and move it according to the moving path planned by the path planning module, so that the endoscope can be withdrawn from the cavity to the outside of the cavity, and then moved to the cleaning basin for cleaning and the wiping cloth for drying. After the blur judgment module determines that the endoscope lens is clear, the endoscope is moved into the cavity; Blur judgment module: used to judge whether the endoscope lens is clear; Wherein, the path planning module includes: Cleaning path planning unit: used to set the node outside the cavity as the starting point and the cleaning basin as the target point, plan the endoscope movement path according to the node position information returned by the depth camera and the position information of the obstacles, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer, control the robotic arm to hold the endoscope to the cleaning basin, and wash the endoscope with water; Wiping path planning unit: used to set the wiping cloth as the starting point and the top of the wiping cloth as the target point, plan the endoscope moving path, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the top of the wiping cloth; Return path planning unit: used to set the top of the wiping cloth as the starting point and the outside of the cavity as the target point, plan the movement path of the endoscope, and then convert it into the movement path of the robotic arm; transmit the planned path to the industrial computer to control the robotic arm to hold the endoscope to the outside of the cavity; Obstacle avoidance path planning module: a solution for obtaining the position of objects using a depth camera, obtaining the positions of the robotic arm, endoscope, and objects in the scene, and dividing the process of cleaning the endoscope into several small processes according to the nodes therein. For each small process, the starting point and end point of the endoscope movement are input; for each small process, the shortest path for the robotic arm to avoid obstacles is planned based on the starting point and end point as well as the obstacle information in the middle according to the existing path planning algorithm; the path is solved into the motion trajectory of the robotic arm, and instructions are sent to control the robotic arm to move; during the movement, the possible collision is judged based on the object position information collected in real time by the depth camera, and the path is corrected; The fuzzy judgment module uses the Sobel operator to extract the horizontal and vertical gradients of the received endoscopic image respectively, and then calculates the variance to obtain the fuzzy score of the image; determines whether the fuzzy score of the image exceeds a threshold, and if so, determines it to be a fuzzy image; when the endoscopic images for 3 consecutive seconds are all determined to be fuzzy images, it is determined that the current endoscope needs to be cleaned.

5. The device for automatically flushing an endoscope for an endoscope holding robot according to claim 4, characterized in that: It also includes an object position acquisition unit, which is used to obtain depth information in the picture according to the depth camera, and transmit the picture and the depth information back to the industrial computer. The industrial computer uses a semantic segmentation algorithm to segment the objects in the picture sent back by the depth camera, and then uses a depth estimation algorithm to estimate the depth of each object. The depth camera position is used as a reference to infer the coordinates of each object in the spatial coordinate system according to the depth information of each object.

6. The device for automatically flushing an endoscope for an endoscope holding robot according to claim 4, characterized in that: It also includes an endoscope recognition module, which is used to use photos in which the endoscope lens appears completely and photos in which the lens does not appear completely as binary classification training samples to train the neural network; when the endoscope exits the cavity, the trained binary classification neural network is used to judge the image returned by the depth camera in real time to determine whether the endoscope lens appears completely in the image; when a complete endoscope lens is detected, a signal is sent that the endoscope has completely exited the cavity, so that the robotic arm can perform the next stage of movement.

Images