A control method and device of a surgical robot system and the surgical robot system

Abstract

The application provides a control method and device of a surgical robot system and the surgical robot system, and relates to the technical field of medical equipment.The control method of the surgical robot system comprises the following steps: acquiring sensing information; when a target face image of a target person wearing surgical glasses exists in a preset intermediate pixel region of a current image, screening an indication mark image meeting preset contour conditions and preset area conditions from the target face image; when the number of the indication mark image is equal to the number of indication marks, obtaining current position information of the indication marks according to the indication mark image; and when the current position information meets preset position distribution conditions, controlling an operating device to enter a master-slave control state.The application can quickly and accurately locate the center points of the eyes of the target person through the indication marks, and the operating device enters the master-slave control state when the target person looks at the center point of the screen, thereby improving the safety of operating the surgical robot system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical equipment, in particular to a control method and device of a surgical robot system and the surgical robot system. BACKGROUND

[0002] With the continuous development of high-end technologies such as robot control technology and image processing technology, surgical robot systems applied to the medical field have emerged as the times require. The surgical robot system is usually composed of an operating device controlled by a doctor, a surgical mechanical arm, a display device and the like, and can assist the doctor to perform fine surgical operations with high precision and high stability.

[0003] The operating device and the surgical mechanical arm of the prior art are generally connected in real time, and the doctor usually observes the display screen while controlling the operating device during the operation. However, in the preparation stage before the doctor performs the operation by operating the device, if the doctor or other medical staff operating the surgical robot system accidentally touches the operating device without looking at the display screen, it is very likely to cause a safety accident. SUMMARY

[0004] The problem solved by the present application is how to improve the safety of operating a surgical robot system.

[0005] To solve the above problems, the present application provides a control method of a surgical robot system, applied to a surgical robot system, the surgical robot system comprising an operating device, a display device, a sensing device and surgical glasses; the sensing device is used for shooting a current image; the surgical glasses are provided with at least three indication marks for determining the center points of the eyes of a target person wearing the surgical glasses; the control method of the surgical robot system comprises:

[0006] obtaining sensing information, the sensing information comprising a current image collected by the sensing device;

[0007] when a target face image of the target person wearing the surgical glasses exists in a preset middle pixel area of the current image, screening an indication mark image meeting preset contour conditions and preset area conditions from the target face image;

[0008] when the number of the indication mark images is equal to the number of the indication marks, obtaining current position information of the indication marks according to the indication mark images;

[0009] when the current position information meets a preset position distribution condition that the target person looks at the center point of the screen of the display device, controlling the operating device to enter a master-slave control state.

[0010] Optionally, the perception device is further configured to measure a current measured distance between the binocular center point and a ranging center point of the perception device; the perception information further comprises a preset fixed distance between the ranging center point and the screen center point and the current measured distance; and the control method of the surgical robot system further comprises:

[0011] when the current position information satisfies the position distribution condition, determining reference information based on the current measured distance and the preset fixed distance, wherein the reference information comprises a reference gaze distance between the binocular center point and the screen center point, and a reference included angle between a line connecting the ranging center point and the binocular center point and a line connecting the ranging center point and the screen center point;

[0012] when the operation device is in the master-slave control state, obtaining a current gaze distance between the binocular center point and the screen center point based on the current measured distance and the reference included angle;

[0013] when a difference between the current gaze distance and the reference gaze distance is greater than a preset difference threshold, controlling the operation device to exit the master-slave control state.

[0014] Optionally, after the difference between the current gaze distance and the reference gaze distance is greater than the preset difference threshold, the control method further comprises:

[0015] when the current position information again satisfies the position distribution condition, controlling the operation device to enter the master-slave control state and updating the reference information based on the current measured distance.

[0016] Optionally, before the target facial image of the target personnel wearing the surgical glasses is screened from the target facial image based on the preset contour condition and the preset area condition, the control method further comprises:

[0017] determining whether a facial image of the target personnel exists in the preset intermediate pixel region of the current image;

[0018] if yes, determining whether the target personnel wears the surgical glasses based on the facial image.

[0019] Optionally, the determination of whether the target personnel wears the surgical glasses based on the facial image comprises:

[0020] graying, binarizing and edging the facial image to obtain a facial contour image;

[0021] determining whether the target person wears the surgical glasses based on the face contour image, including determining whether a number of pixel points in the face contour image is greater than a preset value, wherein the preset value is determined according to the current measurement distance;

[0022] If yes, the face image is taken as the target face image.

[0023] Optionally, when the number of the indication mark images is equal to the number of the indication marks, the current position information of the indication mark is obtained according to the indication mark image, including:

[0024] graying, binarizing, edging, rectangularizing and circle fitting of the indication mark image to obtain an indication mark contour image;

[0025] determining coordinates of a mark center point according to the indication mark contour image to obtain the current position information.

[0026] Optionally, when the number of the indication marks is three, the position distribution condition includes:

[0027] a line connecting the three mark center points forms an isosceles triangle, and an included angle between a base of the isosceles triangle and a horizontal plane is less than a preset angle threshold.

[0028] The present application can master the situation in front of the display device in real time by acquiring the current image and other sensing information sent by the sensing device. The surgical glasses are provided with an indication mark for determining the center point of the target person's eyes wearing the surgical glasses. When the target person wearing the surgical glasses exists in front of the display device, the pupil of the target person does not need to be tracked, and the position of the center point of the target person's eyes can be quickly located according to the position of the indication mark on the surgical glasses in the current image, thereby providing a reliable reference basis for subsequent judgment of whether the target person is looking at the display device. When the target face image of the target person wearing the surgical glasses exists in the preset middle pixel region of the current image, it can be ensured that the center point of the eyes is located in the space range in front of the middle region of the display device, which is the most basic condition for the target person looking at the center point of the screen, and is also beneficial to eliminate the interference of other persons in the current image. At this time, the indication mark image meeting the preset contour condition and the preset area condition is screened out from the target face image, wherein the preset contour condition can exclude other interference information whose shape contour does not conform to the indication mark, and the preset area condition can further exclude the light plate with too large area and the noise point with too small area, thereby being beneficial to guarantee the accuracy and authenticity of the indication mark image. Based on the number of indication mark images, the case that the indication mark images are less than the actual number of indication marks due to the large amplitude rotation of the head of the target person relative to the display device can be quickly excluded, thereby being beneficial to save the computing power and improve the judgment efficiency. When the number of indication mark images is equal to the number of indication marks, the current position information of the indication mark is obtained according to the indication mark image, thereby providing a reference basis for subsequent accurate judgment of whether the center point of the eyes of the target person looks at the center point of the screen. When the head of the target person is skewed or rotated relative to the display device, the positions of at least three indication marks will also change accordingly. When the current position information meets the preset position distribution condition, it can be ensured that the head of the target person does not produce large amplitude skew or rotation relative to the display device, thereby being beneficial to guarantee that the center point of the eyes of the target person looks at the center point of the screen. At this time, the control operation device enters the master-slave control state, thereby avoiding the misoperation caused by the doctor performing the operation without looking at the operation device, and improving the safety of the operation robot system.

[0029] The present application also provides a control device of a surgical robot system, which comprises a computer readable storage medium storing a computer program and a processor. When the computer program is read and run by the processor, the control method of the surgical robot system is realized.

[0030] The control device of the surgical robot system provided by the present application has basically the same advantages as the control method of the surgical robot system compared with the prior art, and will not be described here.

[0031] The application also provides a surgical robot system, comprising an operating device, a display device, a sensing device, surgical glasses and a control device of the surgical robot system as described above.

[0032] The surgical robot system provided by the application has basically the same advantages as the control device of the surgical robot system compared with the prior art, which will not be repeated here.

[0033] The application also provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the control method of the surgical robot system as described above is realized.

[0034] The computer readable storage medium provided by the application has basically the same advantages as the control method of the surgical robot system compared with the prior art, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the control method of the surgical robot system of the embodiment of the application;

[0036] Figure 2 The flowchart of the control method of the surgical robot system of another embodiment of the application;

[0037] Figure 3 The schematic diagram of the depth frame in the current image of the embodiment of the application;

[0038] Figure 4 The position schematic diagram of the binocular center point, the screen center point and the ranging center point when the current position information meets the position distribution condition of the embodiment of the application. DETAILED DESCRIPTION

[0039] In order to make the above objectives, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings. Although some embodiments of the application are shown in the drawings, it should be understood that the application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, on the contrary, these embodiments are provided to make the application more thorough and complete. It should be understood that the drawings and embodiments of the application are only for exemplary purposes, and are not intended to limit the protection scope of the application.

[0040] It should be understood that each step described in the method embodiments of the application can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the application is not limited in this respect.

[0041] The term "include," and derivations thereof, is an open term that means "including, but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optional" means "optional in at least some embodiments." Related definitions are given throughout the detailed description. It is to be noted that the terms "first," "second," and the like are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0042] It should be noted that the terms "one," "multiple," and the like used in the present application are illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as "one or more."

[0043] As shown in Figure 1 An embodiment of the present application provides a control method of a surgical robot system, applied to a surgical robot system, the surgical robot system comprising an operation device, a display device, a sensing device and a surgical glasses; the sensing device is used to shoot a current image; the surgical glasses are provided with at least three indicating marks for determining the center points of the eyes of a target person wearing the surgical glasses; the control method of the surgical robot system comprises the following steps:

[0044] S1: acquiring sensing information, the sensing information comprising a current image collected by the sensing device.

[0045] Specifically, the operation device in the embodiment refers to a device controlled by a doctor to perform surgical operation, such as an operation robot, an operation rod, an operation table integrated with multiple function buttons, etc.; the display device in the embodiment refers to a device (such as a display screen) capable of presenting images in real time during surgery to a target person (i.e., the doctor), and the operation device is usually arranged directly below the display device to facilitate the target person to control the operation device while watching the screen of the display device; the sensing device in the embodiment refers to a device capable of collecting image information in front of the display device, such as a camera installed directly above the display screen; the surgical glasses in the embodiment refer to glasses worn by the target person during surgery, which can be 3D surgical glasses or protective glasses, and the surgical glasses are provided with at least three indicating marks (such as three circular reflective patches), and the position of the indicating marks can determine the position of the center point of the target person's eyes wearing the surgical glasses, for example, three circular labels are attached at the connection between the two lenses of the surgical glasses, and the center of the triangle formed by the centers of the three circular labels is the position of the center point of the eyes. For example, during the use of the surgical robot system, the target person wears the surgical glasses to face the screen of the display device, and the sensing device is arranged directly above the screen of the display device and can collect the current image of the target person in front of the display device. When the surgical robot system is normally used for surgical operation, the target person needs to watch the screen of the display device while controlling the operation device arranged below the display device to perform surgical operation, but when the target person does not look at the screen during the preoperative preparation stage, a safety accident may be caused by accidental touch of the operation device. The execution subject (i.e., the controller) of the embodiment provides an accurate reference for controlling the operation device by receiving the current image and judging whether the target person looks at the center point of the screen of the display device based on the current position information of the indicating marks in the current image, which is beneficial to ensure the safety of the operation of the surgical robot system.

[0046] It should be understood that the looking in the embodiment is not in the absolute sense of completely looking, and a small amplitude rotation or skew of the head of the target person relative to the screen of the display device is allowed. The purpose of the at least three indicating marks in the embodiment is to determine at least three points to facilitate subsequent judgment of whether the target person looks at the display device (i.e., whether the head wearing the surgical glasses is rotated or skewed) according to the positions of the three points, and other shapes and numbers of indicating marks that utilize the same principle to achieve the same effect also belong to the protection scope of the embodiment, for example, the indicating marks can also be a rectangular mark, and whether the target person looks at the display device can also be judged by judging whether the angles of the edges of the rectangular mark change according to the coordinates of the four vertices of the rectangular mark.

[0047] In the embodiment, the situation in front of the display device can be grasped in real time by acquiring the current image and other sensing information sent by the sensing device. The surgical glasses are provided with an indication mark for determining the center point of the eyes of the target person. When the target person wearing the surgical glasses exists in front of the display device, the position of the indication mark on the surgical glasses in the current image can be used to accurately grasp the position of the center point of the eyes of the target person. Compared with the prior art, the method of positioning the center point of the eyes by determining the movement track of the face or the movement track of the pupil is more accurate, convenient and efficient, and provides a reliable reference basis for subsequent judgment of whether the target person is looking at the display device.

[0048] S2: When the target face image of the target person wearing the surgical glasses exists in the preset intermediate pixel region of the current image, an indication mark image meeting preset contour conditions and preset area conditions is selected from the target face image.

[0049] Specifically, the preset intermediate pixel region in the embodiment represents a preset region around the center point in the pixel frame, for example, a pixel region within 200 pixels above, below, left and right of the center point. When the target face image of the target person wearing the surgical glasses exists in the preset intermediate pixel region, the face of the target person can be guaranteed to be within the preset range around the screen center point, which is a basic prerequisite for the target person looking at the screen center point. The preset contour conditions and the preset area conditions in the embodiment can be determined in advance according to the contour and the area of the indication mark. For example, if the indication mark is three circular marks, the preset contour condition is approximately circular, and the preset area condition can be determined according to the imaging rule that large images are far away and small images are close. The area of the indication mark at a distance of 0.3 m and 1 m in front of the display device is obtained in advance to obtain a preset area range. The image with a circular contour and a contour area within the preset area range is retained as an indication mark image, which is helpful to exclude interference with too large or too small area. For example, after excluding images that do not meet the preset contour conditions, the contour area of each image can be obtained by using the contourArea function, and the contour with a pixel value in the range of 370-400 can be selected to exclude larger circular light spots and smaller noise points caused by strong light.

[0050] In the embodiment, when the target face image of the target personnel wearing the surgical glasses exists in the preset intermediate pixel region of the current image, the face position of the target personnel can be guaranteed to be within a preset range around the screen center point when projected to the display device, which is a basic prerequisite for the target personnel looking at the screen center point. On this basis, the indicator image meeting the preset outline condition and the preset area condition is screened out from the target face image, wherein the preset outline condition can exclude other interference information with a shape outline not conforming to the indicator, and the preset area condition can further exclude face structure interference with an area being too large and noise with an area being too small, which is beneficial to guarantee the accuracy and authenticity of the indicator image.

[0051] S3: When the number of the indicator images is equal to the number of the indicators, current position information of the indicators is obtained according to the indicator images.

[0052] Specifically, when the head of the target personnel wearing the surgical glasses rotates by a large amplitude relative to the display device, the indicator that can be photographed by the perception device may be reduced. For example, assuming that the number of the indicators is 3 and the indicators are arranged in sequence along the horizontal direction, when the head of the target personnel rotates by too large an angle to the left, the leftmost indicator may not be photographed by the perception device, and the number of the indicator images is less than the actual number of the indicators, which indicates that the head of the target personnel rotates by a large amplitude and does not look at the screen center point. When the number of the indicator images is equal to the number of the indicators, it can be guaranteed that the head of the target personnel does not rotate by a large amplitude relative to the display device (i.e., the head of the target personnel does not rotate by a large angle around the vertical direction). On this basis, the current position information of the indicators obtained according to the indicator images can provide a basis for further judging whether the target personnel looks at the screen center point. The current position information of the indicators can be obtained through the indicator images. For example, the pixel frame of the indicator image is obtained, and the coordinate information of the indicator in the pixel frame is determined according to the position of the indicator in the pixel frame, to obtain the current position information.

[0053] In the embodiment, when the number of the indicator images is equal to the number of the indicators, it can be guaranteed that the head of the target personnel does not rotate by a large amplitude relative to the display device, and on this basis, the current position information of the indicators obtained according to the indicator images can provide a reference for further judging whether the center points of the eyes of the target personnel look at the screen center point.

[0054] S4: When the current position information meets the preset position distribution condition that the target personnel looks at the screen center point of the display device, the operation device is controlled to enter a master-slave control state.

[0055] Specifically, the preset position distribution condition corresponding to the target person looking at the center point of the screen in the embodiment indicates that the line connecting the center points of the eyes of the target person and the center line point of the screen is approximately perpendicular to the screen of the display device, and the corresponding position distribution of the indicating mark is indicated. Since when the head of the target person is skewed or rotated relative to the display device, the positions of the at least three indicating marks will also change accordingly, based on this, whether the target person is looking at the center point of the screen can be judged through the current position information of the indicating mark. For example, when the indicating mark is three circular marks arranged equidistantly along the horizontal direction, the center coordinates of the three circular marks are obtained according to the current position information, which are respectively the first coordinate (x1, y1), the second coordinate (x2, y2) and the third coordinate (x3, y3). When the head of the target person is skewed to the right, y1>y2>y3 will appear, if the difference between y1 and y3 is greater than a first preset difference, it is determined that the head of the target person is skewed greatly, and the preset position distribution condition is not met; when the head of the target person is rotated, (x1+x3) / 2>x2 will appear, or (x1+x3) / 2<x2 will appear, but the difference between (x1+x3) / 2 and x2 is greater than a second preset difference, which indicates that the head of the target person is rotated greatly, and the preset position distribution condition is not met. Based on this, whether the target person is looking at the display screen can be judged; when the difference between y1 and y3 is less than the first preset difference, and the difference between (x1+x3) / 2 and x2 is less than the second preset difference, the current position meets the preset position distribution condition, which indicates that the head of the target person does not have a large rotation or skew relative to the screen of the display device, and the control operation device enters the master-slave control state, avoiding safety accidents caused by the doctor operating the operation device without looking at it.

[0056] The embodiment can master the situation in front of the display device in real time by acquiring the current image and other sensing information sent by the sensing device. The surgical glasses are provided with an indication mark for determining the center point of the target person's eyes wearing the surgical glasses. When the target person wearing the surgical glasses exists in front of the display device, the pupil of the target person does not need to be tracked, and the position of the center point of the target person's eyes can be quickly located according to the position of the indication mark on the surgical glasses in the current image, thereby providing a reliable reference basis for subsequent judgment of whether the target person is looking at the display device. When the target face image of the target person wearing the surgical glasses exists in the preset middle pixel region of the current image, it can be ensured that the center point of the eyes is located in the space range in front of the middle region of the display device, which is the most basic condition for the target person looking at the center point of the screen, and is also beneficial to eliminate the interference of other persons in the current image. At this time, the indication mark image meeting the preset contour condition and the preset area condition is screened out from the target face image, wherein the preset contour condition can exclude other interference information whose shape contour does not conform to the indication mark, and the preset area condition can further exclude the light plate with too large area and the noise point with too small area, which is beneficial to ensure the accuracy and authenticity of the indication mark image. Based on the number of indication mark images, the case that the indication mark images are less than the actual number of indication marks due to the large amplitude rotation of the head of the target person relative to the display device can be quickly excluded, which is beneficial to save computing power and improve the judgment efficiency. When the number of indication mark images is equal to the number of indication marks, the current position information of the indication mark is obtained according to the indication mark image, which can provide a reference basis for subsequent accurate judgment of whether the center point of the target person's eyes looks at the center point of the screen. When the head of the target person is skewed or rotated relative to the display device, the positions of at least three indication marks will also change accordingly. When the current position information meets the preset position distribution condition, it can be ensured that the head of the target person does not produce large amplitude skew or rotation relative to the display device, which is beneficial to ensure that the center point of the target person's eyes looks at the center point of the screen. At this time, the control operation device enters the master-slave control state, avoids the misoperation caused by the doctor performing surgical operation without looking at the operation device, and improves the safety of the operation robot system.

[0057] Optionally, as shown in Figure 2 The sensing device is further configured to measure a current measurement distance between the center point of the eyes and a ranging center point of the sensing device; the sensing information further includes a preset fixed distance between the ranging center point and the center point of the screen and the current measurement distance; and the control method of the surgical robot system further includes:

[0058] When the current location information meets the location distribution conditions, reference information is determined based on the current measurement distance and the preset fixed distance. The reference information includes the reference gaze distance between the center point of both eyes and the center point of the screen, and the reference angle between the line connecting the distance measurement center point and the center point of both eyes and the line connecting the distance measurement center point and the center point of the screen.

[0059] When the operating device is in master-slave control mode, the current gaze distance between the center point of both eyes and the center point of the screen is obtained based on the current measurement distance and the reference angle.

[0060] When the difference between the current gaze distance and the reference gaze distance is greater than a preset difference threshold, the control operation device exits the master-slave control state.

[0061] Specifically, the sensing device in this embodiment may further include a device capable of measuring the current distance between the center points of both eyes and the distance measuring center point of the sensing device, such as an infrared rangefinder. Preferably, the sensing device in this embodiment uses a depth camera. The current image can be obtained based on the color frame of the current image, and the current measurement distance can be obtained based on the depth frame of the current image, wherein the distance measuring center point is the center point of the depth camera lens. Optionally, to facilitate accurate acquisition of the current measurement distance, a distance measuring marker can be set at the position corresponding to the center points of both eyes on the surgical glasses for the depth camera to identify. Optionally, such as... Figure 3 As shown, when the target person wears surgical glasses, the indicators on the glasses may obstruct the area where the center points of the eyes are located, preventing the depth camera from directly obtaining the distance between the center points of the eyes and the distance measurement center point. In this case, depth information corresponding to at least three indicators can be obtained separately, and the distance between the center points of the eyes and the distance measurement center point can be determined based on the positional distribution of the at least three indicators. Optionally, the current measurement distance of other positions on the target person's face can also be obtained based on the depth frame in the current image. For example, the tip of the target person's nose can be set as the region of interest (corresponding to...). Figure 3 (Select the area in the box), measure the distance between the point and the distance measurement center point, and then combine the fixed distance from the point to the center point of both eyes to obtain the current distance measurement between the center point of both eyes and the distance measurement center point.

[0062] In an embodiment, the sensing device can be installed above the screen of the display device. Since the positions of the sensing device and the display device are relatively fixed, the position and distance relationship between the ranging center point and the screen center point can be obtained in advance to obtain a preset fixed distance. Preferably, the line connecting the ranging center point and the screen center point is perpendicular to the horizontal plane (i.e., the sensing device is installed directly above the display device). When the target person is looking at or deviating from the display screen of the display device, the pose state (such as distance, rotation angle, skew angle, etc.) of the head of the target person relative to the display device will be different, which means that the distance between the eye center point and the screen center point will also change accordingly when the target person is looking at or deviating from the display screen of the display device. When the current position information satisfies the position distribution condition, a triangle is formed between the eye center point, the screen center point, and the measurement center point. Preferably, to reduce the consumption of computing power, the screen center point and the measurement center point are both on the plane of the screen of the display device, and the line connecting the eye center point and the screen center point is perpendicular to the line connecting the eye center point and the screen center point, i.e., the triangle formed between the eye center point, the screen center point, and the measurement center point is a right triangle.

[0063] As Figure 4As shown, in an embodiment, for the convenience of understanding and expression, the embodiment records the measurement center point as point A, the screen center point as point B, the center point of the two eyes as point C, the line between point A and point B as side c, the line between point B and point C as side a, the line between point C and point A as side b, the included angle between side a and side c as θ1, the included angle between side a and side b as θ2, and the included angle between side b and side c as θ3. When the current position information satisfies the position distribution condition, θ1 = 90°, the line between point A, point B and point C forms a right triangle, which is recorded as a reference triangle. Given the size of side b and side c, the size of side a, i.e., the reference gaze distance, can be obtained by solving the triangle (such as using the Pythagorean theorem), and given the length of the three sides of the right triangle, the angle θ2 can be obtained by solving the triangle (such as using the sine theorem), i.e., the reference included angle. When the operating device is in the master-slave control state, the center point of the two eyes of the target person may move in this process, but overall it can be regarded as the center point of the two eyes moving along the circumcircle of the reference triangle. Since the pre-set fixed distance between the ranging center point and the screen center point does not change (i.e., the length of side c does not change), it can be considered that the angle θ2 of side c does not change during the movement of the center point of the two eyes. At this time, given the size of the current side b, the size of side c, and the angle θ2, the current gaze distance (i.e., the length of side a) can be obtained by solving the triangle. When the deviation between the current gaze distance and the reference gaze distance is large, it means that the position of the center point of the two eyes has deviated greatly compared to the screen center point, and the line of sight of the target person may have deviated greatly, and no longer looks at the screen center point. At this time, it may represent that the attention of the target person is distracted, and if the target person continues to control the operating device to perform surgery in this case, many safety accidents may occur, and the master-slave needs to be disconnected, i.e., the control operating device exits the master-slave control state. For example, the reference gaze distance is 0.8 m, and the corresponding difference threshold is 0.1 m, which means that the current gaze distance is within the range of 0.7 m-0.9 m, and it is considered that the target person still looks at the display device. When the current gaze distance is less than 0.7 m or greater than 0.9 m, it is considered that the deviation occurs, and the control operating device exits the master-slave control state.

[0064] In the embodiment, when the current position information meets the position distribution condition, the reference information such as the reference gaze distance and the reference included angle can be determined based on the current measured distance and the preset fixed distance. When the operation device is in the master-slave control state, the binoic center point position will change, and the current gaze distance can be obtained according to the current gaze distance and the reference included angle. When the difference between the current gaze distance and the reference gaze distance is greater than the preset difference threshold, it indicates that the line of sight of the target person is likely to deviate greatly, and the target person no longer looks at the display device. The embodiment does not need to perform complex processing on the current image after determining the reference information, and can directly determine whether the current gaze distance between the binoic center point of the target person and the screen center point changes greatly based on the current measured distance, and further determine whether the line of sight of the target person deviates greatly, thereby improving the disposal efficiency of the system. When the difference between the current gaze distance and the reference gaze distance is greater than the preset difference threshold, the operation device is controlled to exit the master-slave control state, which is beneficial to avoid the misoperation caused by the target person continuing to control the operation device when the line of sight deviates greatly, and ensures the safety of the operation process.

[0065] Optionally, as shown in Figure 2 When the difference between the current gaze distance and the reference gaze distance is greater than the preset difference threshold, after the operation device is controlled to exit the master-slave control state, the method further includes:

[0066] When the current position information meets the position distribution condition again, the operation device is controlled to enter the master-slave control state, and the reference information is updated based on the current measured distance.

[0067] In the embodiment, after the operation device is controlled to exit the master-slave control state, the target person may look at the screen center point again, and the current position information will meet the position distribution condition again. At this time, the operation device can be restored to enter the master-slave control state and continue the operation. Meanwhile, after the target person deviates from the screen of the display device, the target person may adjust the sitting posture, position or even replace the target person for operation. At this time, the reference information is updated according to the current measured distance, which can ensure the accuracy of judging whether the target person looks at the screen center point according to the reference information, improve the adaptability of the system to different target persons and different states of the target person, and ensure the rationality of the control.

[0068] Optionally, as shown in Figure 2 When the target face image of the target person wearing the surgical glasses exists in the preset middle pixel region of the current image, before the indication mark image meeting the preset contour condition and the preset area condition is selected from the target face image, the method further includes:

[0069] determining whether the face image of the target person exists in the preset middle pixel region of the current image;

[0070] If yes, it is determined whether the target personnel wears surgical glasses based on the target face image.

[0071] Specifically, the face image in the preset middle pixel region of the current image is probably the face image of the target personnel performing the surgical operation in front of the display device, and based on this, it is determined whether the target personnel in front of the display device wears surgical glasses, and it is further determined whether the personnel corresponding to the face image is the target personnel performing the surgical operation. For example, after obtaining the current image, the face frame coordinates of the target personnel are obtained by using the face detection function in OPENVINO. If the face frame coordinates are all in the preset middle pixel region (such as within 200 pixels to the left and right of the middle line of the pixel frame), it indicates that there is a face image of the target personnel in the preset middle pixel region. The face image can be taken as a target face image, and it is further determined whether the target personnel wears surgical glasses based on the target face image. Preferably, the processing of the current image can be combined with the current measurement distance obtained by the depth camera, for example, only the face frame coordinates corresponding to the face of the personnel whose current distance is less than 1 m are extracted, which is beneficial to exclude the interference of other remote workers and further reduce the data processing amount.

[0072] In the embodiment, determining whether there is a face image of the target personnel in the preset middle pixel region is beneficial to exclude interference factors, avoid misjudging the active personnel in the remaining range as the target personnel performing the surgical operation, and also reduce the data processing amount and improve the processing efficiency. When there is a face image of the target personnel in the preset middle pixel region, it is determined whether the target personnel wears surgical glasses based on the target face image, and it is further identified whether the identity of the personnel is the target personnel performing the surgical operation.

[0073] Optionally, as shown in Figure 2 determining whether the target personnel wears surgical glasses based on the face image includes:

[0074] graying, binarizing and edge processing the face image to obtain a face contour image;

[0075] determining whether the target personnel wears surgical glasses based on the face contour image includes determining whether the number of pixel points in the face contour image is greater than a preset value, wherein the preset value is determined according to the current measurement distance;

[0076] If yes, the face image is taken as a target face image.

[0077] Specifically, the corresponding face contour images are different when the target personnel wears surgical glasses and does not wear surgical glasses, and the number of pixel points of the corresponding face contour images is also different, so that whether the target personnel wears surgical glasses can be determined based on this. For example, the face image in the RGB format in the color frame is grayed, the pixel value higher than the preset threshold is set to 0, and the pixel value lower than the preset value is set to 255, thereby obtaining a binary image, which is beneficial to screen out high-frequency and low-frequency information that needs to be paid attention to. On this basis, the contour is outlined according to the white pixel area of the binary image, the edge information is extracted, and the face contour image is obtained. The number of white pixel points is extracted based on the face contour image, and whether the target personnel wears surgical glasses is determined in combination with the preset value. For example, the current measurement distance is 0.8 m, the preset value of the corresponding pixel points is 1200, the number of extracted pixel points is 1231, and it is indicated that the target personnel wears surgical glasses, so it can be determined that the target personnel is the target personnel performing surgical operation, and the corresponding face image is taken as the target face image.

[0078] In the embodiment, the face image is grayed, binarized and edge-processed to obtain a face contour image, which is beneficial to quickly extract high-frequency and low-frequency information that needs to be paid attention to, and can also reduce the difficulty of data processing. When the number of pixel points in the face contour image is greater than the preset value, it indicates that the target personnel wears glasses, and it can be determined that the target personnel is the target personnel performing surgical operation, and the corresponding face image is taken as the target face image. Since the target personnel is at different distances from the screen of the display device, the number of pixel points corresponding to the face contour is different, and the preset value in the embodiment is determined according to the current measurement distance, which is beneficial to improve the accuracy of the determination result.

[0079] Optionally, as shown in Figure 2 When the number of indication mark images is equal to the number of indication marks, the current position information of the indication mark is obtained according to the indication mark image, including:

[0080] The indication mark image is grayed, binarized, edge-processed, rectangularized and fitted into a circle to obtain an indication mark contour image.

[0081] The coordinates of the mark center point are determined according to the indication mark contour image to obtain the current position information.

[0082] In the embodiment, the instruction mark image is subjected to grayscale processing, binarization, and edge processing to obtain a contour image. Preferably, the infrared frame of the instruction mark image captured by the depth camera is selected, and the instruction mark contour image can be quickly obtained by only performing binarization, edge processing, rectangularization, and circle fitting on the infrared frame. On this basis, the instruction mark contour image can be obtained by performing rectangularization and then fitting a circle (e.g., using the least square method) on the instruction mark contour image. The points on the instruction mark contour image correspond to different pixel positions. The center point of the pixel frame is set as the origin, and a rectangular coordinate system is established, so that the coordinate information of the instruction mark can be obtained. Based on this, the coordinates of the center point of each instruction mark contour image can be determined to obtain the current position information. This provides a precise and reliable reference for subsequent determination of whether the target person is looking at the center point of the screen based on the current position information.

[0083] Optionally, when the number of instruction marks is three, the position distribution condition comprises:

[0084] The line connecting the three mark center points forms an isosceles triangle, and the included angle between the base of the isosceles triangle and the horizontal plane is less than a preset angle threshold.

[0085] In the embodiment, the preset angle threshold can be adjusted according to actual conditions. The greater the preset angle threshold, the greater the degree of tolerance for the target person's head tilt, and vice versa. In the embodiment, the value range of the preset angle threshold is 5-15°, and preferably, the preset angle threshold is 10°. In the embodiment, the line connecting the three center points of the three indication marks can be arranged in an isosceles triangle on the surgical glasses. For the sake of understanding and description, the three center points are respectively denoted as a vertex, a first bottom vertex and a second bottom vertex. The coordinates of the center points are determined according to the indication mark contour image. When the line connecting the three center points forms an isosceles triangle, it is ensured that the target person's head does not rotate by a large amplitude. For example, the coordinates corresponding to the vertex, the first bottom vertex and the second bottom vertex are obtained respectively, the length between the vertex and the first bottom vertex is obtained based on the coordinates, and denoted as a first length, the length between the vertex and the second bottom vertex is obtained based on the coordinates, and denoted as a second length. When the difference between the first length and the second length is less than a third preset difference (such as 0.5 cm), it is considered that the line connecting the three center points forms an isosceles triangle. It should be understood that the line connecting the three center points in the embodiment forms an isosceles triangle in the absolute sense, but allows a certain range of deviation between the first length and the second length, which is more suitable for actual application. On this basis, the angle between the base of the isosceles triangle and the horizontal plane is less than the preset angle threshold, which further ensures that the target person's head does not tilt by a large amplitude. For example, the coordinates of the first bottom vertex and the second bottom vertex are known, and the angle between the base and the horizontal plane can be determined according to the difference between the horizontal and vertical coordinates. When the angle is less than the preset angle threshold, it indicates that the target person's head does not tilt by a large amplitude. In this way, it can be ensured that the plane where the isosceles triangle is located is substantially parallel to the screen of the display device, and the angle between the plane and the horizontal plane is within the allowable range, which is beneficial to ensure that the target person looks at the screen center point, and realizes that the target person looks at the screen center point based on the current position information of the indication mark.

[0086] Another embodiment of the present application provides a control device of a surgical robot system, comprising a computer readable storage medium storing a computer program and a processor, and when the computer program is read and run by the processor, the control method of the surgical robot system is realized.

[0087] The control device of the surgical robot system provided in the embodiment and the control method of the surgical robot system can achieve basically the same technical effects, which will not be described here.

[0088] Another embodiment of the present application provides a surgical robot system, comprising an operating device, a display device, a sensing device, a surgical glasses and a control device of a surgical robot system as described above.

[0089] The surgical robot system provided by the embodiments and the technical effects that can be achieved by the control device of the surgical robot system are basically the same, and thus will not be described herein again.

[0090] Another embodiment of the present application provides a computer readable storage medium, and the computer program is stored on the storage medium. When the computer program is executed by a processor, the control method of the surgical robot system is realized.

[0091] The computer readable storage medium provided by the embodiments and the technical effects that can be achieved by the control method of the surgical robot system are basically the same, and thus will not be described herein again.

[0092] An electronic device that can be a server or a client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent a wide variety of digital electronic computing devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent a wide variety of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components, their connections, and their functions, as described herein, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0093] The electronic device includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded into a random access memory (RAM) from a storage unit. In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0094] The computer system can include clients and servers. A client and a server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like. In this application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0096] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A control method of a surgical robot system, characterized by, The application is applied to a surgical robot system, the surgical robot system comprises an operation device, a display device, a sensing device and surgical glasses; the sensing device is used for shooting a current image and measuring a current measured distance between a binocular center point and a ranging center point of the sensing device; at least three indication marks for determining a binocular center point of a target personnel wearing the surgical glasses are arranged on the surgical glasses; and a control method of the surgical robot system comprises the following steps: obtaining sensing information, the sensing information comprising a current image collected by the sensing device, a preset fixed distance between the ranging center point and a screen center point, and the current measured distance; when a target face image of the target personnel wearing the surgical glasses exists in a preset middle pixel region of the current image, screening an indication mark image meeting preset outline conditions and preset area conditions from the target face image; when the number of the indication mark image is equal to the number of the indication marks, obtaining current position information of the indication marks according to the indication mark image; when the current position information meets a preset position distribution condition that the target personnel looks at the screen center point of the display device, controlling the operation device to enter a master-slave control state; when the current position information meets the position distribution condition, determining reference information based on the current measured distance and the preset fixed distance, wherein the reference information comprises a reference gaze distance between the binocular center point and the screen center point, and a reference included angle between a line connecting the ranging center point and the binocular center point and a line connecting the ranging center point and the screen center point; when the operation device is in the master-slave control state, obtaining a current gaze distance between the binocular center point and the screen center point based on the current measured distance and the reference included angle; when a difference between the current gaze distance and the reference gaze distance is greater than a preset difference threshold, controlling the operation device to exit the master-slave control state.

2. The control method of the surgical robot system according to claim 1, characterized by, After the step of when the difference between the current gaze distance and the reference gaze distance is greater than the preset difference threshold, controlling the operation device to exit the master-slave control state, the method further comprises the following steps: when the current position information meets the position distribution condition again, controlling the operation device to enter the master-slave control state, and updating the reference information based on the current measured distance.

3. The control method of the surgical robot system according to claim 1, characterized by, Before the step of when the target face image of the target personnel wearing the surgical glasses exists in the preset middle pixel region of the current image, screening the indication mark image meeting the preset outline conditions and the preset area conditions from the target face image, the method further comprises the following steps: determining whether a face image of the target personnel exists in the preset middle pixel region of the current image; if yes, determining whether the target personnel wears the surgical glasses based on the face image.

4. The control method of the surgical robot system according to claim 3, characterized by, The step of determining whether the target personnel wears the surgical glasses based on the face image comprises the following steps: performing gray-scale processing, binary processing and edge processing on the face image to obtain a face outline image; The target personnel is determined to wear the surgical glasses or not based on the face contour image, including determining whether the number of pixel points in the face contour image is greater than a preset value, wherein the preset value is determined according to the current measurement distance. If yes, the face image is taken as the target face image.

5. The control method of the surgical robot system according to any one of claims 1 to 4, characterized in that, When the number of the indication mark images is equal to the number of the indication marks, the current position information of the indication marks is obtained according to the indication mark images, including: The indication mark images are processed by graying, binarizing, edging, rectangularizing and fitting circle to obtain indication mark contour images; The coordinates of the mark center points are determined according to the indication mark contour images to obtain the current position information.

6. The control method of the surgical robot system according to claim 5, characterized by, When the number of the indication marks is three, the position distribution condition includes: The line connecting the three mark center points forms an isosceles triangle, and the included angle between the base of the isosceles triangle and the horizontal plane is less than a preset angle threshold.

7. A control device of a surgical robot system, characterized by, The computer readable storage medium storing the computer program and the processor, when the computer program is read and run by the processor, realize the control method of the surgical robot system as claimed in any one of claims 1 to 6.

8. A surgical robotic system, characterized by, The control device of the surgical robot system, including an operating device, a display device, a sensing device, surgical glasses and the control device as claimed in claim 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores the computer program, and when the computer program is executed by the processor, the control method of the surgical robot system as claimed in any one of claims 1 to 6 is realized.

Images