Sight detection device, surgical robot system, control method and storage medium

Abstract

The present invention provides a line of sight detection device, a surgical robot system, a control method and a storage medium, wherein the line of sight detection device includes a head-mounted device and a controller that are communicatively connected: the head-mounted device is used to detect the head posture information of the subject and transmit the head posture information to the controller; the controller is used to determine whether the subject's line of sight is facing a display screen based on the head posture information and preset conditions. Compared with the method of realizing line of sight detection by collecting eye movement data, the present invention has lower requirements on the position stability of the subject during the line of sight detection process, and the line of sight detection will not be completed because the subject deviates from the predetermined test range for some reason, thereby greatly improving the operability of the line of sight detection. In addition, when the present invention determines that the doctor's line of sight is not facing the display screen, it will send an alarm signal and / or activate a protection mechanism, thereby improving the safety during the operation and effectively preventing misoperation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a sight line detection device, a surgical robot system, a control method and a storage medium. Background Art

[0002] Existing medical devices generally track a doctor's gaze by collecting eye movement data. Common eye trackers use infrared irradiation devices to illuminate the doctor's eyes to generate light spots on the doctor's cornea, and then use these light spots and the doctor's pupil image to calculate the doctor's gaze direction. However, the use of an eye tracker for line of sight detection is greatly interfered with by interference factors such as ambient light. In addition, since the shooting range of the camera installed on the eye tracker for capturing eye movements is limited, there are certain requirements for the position stability of the doctor during the line of sight detection process. Once the doctor deviates from the predetermined test range for some reason, the doctor's line of sight detection will not be successfully completed. With respect to the problem that the above-mentioned existing technologies cannot accurately determine the doctor's line of sight, no effective solution has yet been proposed. Summary of the Invention

[0003] The purpose of the present invention is to provide a line of sight detection device, a surgical robot system, a control method and a storage medium, which can solve the problem in the prior art that the doctor's line of sight cannot be accurately determined.

[0004] To solve the above technical problems, the present invention provides a sight line detection device, comprising a head-mounted device and a controller in communication connection;

[0005] The head-mounted device is used to detect head posture information of the subject and transmit the head posture information to the controller;

[0006] The controller is used to determine whether the subject's line of sight is facing the display screen according to the head posture information and preset conditions.

[0007] Optionally, the head posture information includes horizontal head orientation information and / or vertical head orientation information, and the controller is used to determine whether the horizontal head orientation information meets a first preset condition and / or whether the vertical head orientation information meets a second preset condition. If the horizontal head orientation information does not meet the first preset condition and / or the vertical head orientation information does not meet the second preset condition, the controller instructs to issue an alarm signal and / or activate a protection mechanism.

[0008] Optionally, the head-mounted device includes at least one first magnetic sensor communicatively connected to the controller, the first magnetic sensor being used to detect the angle between the horizontal orientation of the subject's head and a predetermined magnetic field to obtain the horizontal orientation information of the head, and / or being used to detect the angle between the vertical orientation of the subject's head and the predetermined magnetic field to obtain the vertical orientation information of the head.

[0009] Optionally, the controller is used to determine whether the angle between the vertical orientation of the subject's head and the predetermined magnetic field is within a second preset range. If not, it indicates to send an alarm signal and / or start a protection mechanism.

[0010] Optionally, the head-mounted device is further used to detect head movement information of the subject and transmit the head movement information to the controller;

[0011] The controller is further configured to determine whether the head movement information satisfies a third preset condition, and if not, instruct to issue an alarm signal and / or activate a protection mechanism.

[0012] Optionally, the head-mounted device further includes an acceleration sensor in communication with the controller, wherein the acceleration sensor is used to detect the acceleration of the subject's head movement;

[0013] The controller is also used to integrate the head movement acceleration to obtain the head movement distance of the subject, and to determine whether the head movement distance is within a third preset range. If not, an alarm signal is issued and / or a protection mechanism is activated.

[0014] Optionally, the head-mounted device further includes a gyroscope in communication with the controller, wherein the gyroscope sensor is used to detect a vertical posture vector of the head of the subject;

[0015] The controller is further configured to determine whether the head vertical posture vector is within a fourth preset range, and if not, to indicate that an alarm signal is issued and / or a protection mechanism is activated.

[0016] In order to solve the above technical problems, the present invention also provides a surgical robot system, including a display screen, a robotic arm and a line of sight detection device as described above, wherein the display screen is communicatively connected to the controller, and the controller is used to: if it is determined that the line of sight of the subject is not facing the display screen, then control the surgical robot system to send an alarm signal and / or the robotic arm to be in a locked state.

[0017] Optionally, the display screen is provided with at least one second magnetic sensor in communication with the controller, and the second magnetic sensor is used to detect an angle between a screen direction of the display screen and a predetermined magnetic field;

[0018] The controller is used to determine whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range. If not, the controller controls the surgical robot system to issue an alarm signal and / or puts the robotic arm into a locked state.

[0019] To solve the above technical problems, the present invention further provides a control method for a surgical robot system, wherein the surgical robot system includes a display screen and a robotic arm, and the control method includes:

[0020] Obtaining the subject's head posture information;

[0021] Determining whether the head posture information meets a preset condition;

[0022] If yes, it is determined that the subject's line of sight is facing the display screen;

[0023] If not, an alarm signal is issued and / or the robotic arm is controlled to be in a locked state.

[0024] Optionally, obtaining the subject's head posture information includes:

[0025] Acquiring horizontal head orientation information and / or vertical head orientation information of the subject;

[0026] The determining whether the head posture information meets a preset condition includes:

[0027] It is determined whether the horizontal head orientation information satisfies a first preset condition and / or it is determined whether the vertical head orientation information satisfies a second preset condition.

[0028] Optionally, obtaining the subject's head horizontal orientation information and / or head vertical orientation information includes:

[0029] The angle between the horizontal orientation of the subject's head and the predetermined magnetic field and / or the angle between the vertical orientation of the subject's head and the predetermined magnetic field is obtained.

[0030] Optionally, the control method further includes:

[0031] Acquiring an angle between a screen direction of the display screen and the predetermined magnetic field;

[0032] The determining whether the head horizontal orientation information satisfies a first preset condition includes:

[0033] It is determined whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range.

[0034] Optionally, determining whether the head vertical orientation information satisfies a second preset condition includes:

[0035] It is determined whether the angle between the vertical direction of the subject's head and the predetermined magnetic field is within a second preset range.

[0036] Optionally, the control method includes:

[0037] Obtaining the subject's head movement information;

[0038] Determining whether the head movement information meets a third preset condition;

[0039] If not, an alarm signal is issued and / or the robotic arm is controlled to be in a locked state.

[0040] Optionally, obtaining the subject's head movement information includes:

[0041] Obtain the head movement distance of the subject;

[0042] The determining whether the head movement information satisfies a third preset condition includes:

[0043] Determine whether the head movement distance is within a third preset range.

[0044] Optionally, obtaining the head movement distance of the subject includes:

[0045] Obtain the subject's head movement acceleration;

[0046] The head movement acceleration is integrated to obtain the head movement distance of the subject.

[0047] Optionally, obtaining the subject's head movement information includes:

[0048] Obtain the vertical posture vector of the subject's head;

[0049] The determining whether the head movement information satisfies a third preset condition includes:

[0050] Determine whether the head vertical posture vector is within a fourth preset range.

[0051] In order to solve the above technical problems, the present invention also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the control method of the surgical robot system described above is implemented.

[0052] Compared with the prior art, the sight line detection device, surgical robot system, control method, and storage medium provided by the present invention have the following advantages:

[0053] The present invention obtains the head posture information of the subject and determines whether the head posture information meets the preset conditions, thereby determining whether the subject is facing the display screen. Therefore, compared with the method of realizing line of sight detection by collecting eye movement data, the present invention has lower requirements on the position stability of the subject during the line of sight detection process, and will not fail to complete the line of sight detection because the subject deviates from the predetermined test range for some reason, thereby greatly improving the operability of the line of sight detection. In addition, when the present invention determines that the doctor's (subject's) line of sight is not facing the display screen, it will automatically send an alarm signal and / or activate a protection mechanism to lock the robotic arm used to perform the operation, thereby effectively preventing the doctor from operating the robotic arm when his line of sight is not facing the display screen, further improving the safety during the operation and effectively preventing misoperation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a schematic block diagram of a sight line detection device according to an embodiment of the present invention, which also shows a display screen capable of information interaction with the sight line detection device;

[0055] Figure 2 Schematic diagram of the measurement principle of head horizontal orientation information in one embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the measurement principle of head vertical orientation information in one embodiment of the present invention;

[0057] Figure 4 This is a front view of a display screen in one embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the installation of an acceleration sensor on a head-mounted device in one embodiment of the present invention;

[0059] Figure 6 Schematic diagram of the measurement principle of an acceleration sensor in one embodiment of the present invention;

[0060] Figure 7 Schematic diagram of the installation of a gyroscope on a head-mounted device in one embodiment of the present invention;

[0061] Figure 8 Schematic diagram of the measurement principle of a gyroscope in one embodiment of the present invention;

[0062] Figure 9 Schematic diagram of the overall structure of a surgical robot system in one embodiment of the present invention;

[0063] Figure 10 This is a schematic structural diagram of a robotic arm of an operating table in one embodiment of the present invention;

[0064] Figure 11 Schematic diagram of the structure of the doctor's console in one embodiment of the present invention;

[0065] Figure 12 4 is a flow chart of a control method for a surgical robot system in one embodiment of the present invention.

[0066] The accompanying drawings are numerals as follows:

[0067] Head-mounted device 110; controller 120; display screen 130; first magnetic sensor 111; second magnetic sensor 131; acceleration sensor 112; gyroscope 113; first data transceiver 114; second data transceiver 132; doctor's console 10; operating table 20; side trolley 30; robotic arm 31; master control arm 11. DETAILED DESCRIPTION

[0068] The following is combined with Figures 1 to 12 The line of sight detection device, surgical robot system, control method and storage medium proposed in the present invention are further described in detail in the following embodiments. The advantages and features of the present invention will become clearer according to the following description. It should be noted that the drawings are in a very simplified form and are not in precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions for the implementation of the present invention, so they have no technical significance. Any modification of the structure, change in the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0069] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0070] The core idea of ​​the present invention is to provide a line of sight detection device, a surgical robot system, a control method and a storage medium to solve the problem in the prior art that the doctor's line of sight cannot be accurately determined.

[0071] To realize the above idea, the present invention provides a sight line detection device, please refer to Figure 1 , which schematically shows a block diagram of a sight line detection device provided by an embodiment of the present invention, which also shows a display screen that can interact with the sight line detection device. Figure 1 As shown, the gaze detection device includes a connected head-mounted device 110 and a controller 120; the head-mounted device 110 is used to detect the subject's head posture information and transmit the head posture information to the controller 120; the controller 120 is used to determine whether the subject's line of sight is facing the display screen 130 based on the head posture information and preset conditions. Therefore, compared with the method of achieving gaze detection by collecting eye movement data, the gaze detection device provided by the present invention has lower requirements for the subject's positional stability during the gaze detection process. The gaze detection will not be completed due to the subject deviating from the predetermined test range for some reason, greatly improving the operability of the gaze detection.

[0072] To facilitate data transmission, the head-mounted device 110 is provided with a first data transceiver 114. Thus, data transmission between the head-mounted device 110 and the controller 120 can be more easily achieved through the first data transceiver 114. Preferably, the first data transceiver 114 is a Bluetooth module, thereby enabling data transmission between the head-mounted device 110 and the controller 120 via Bluetooth. It should be noted that, as will be understood by those skilled in the art, the first data transceiver 114 may also be other components capable of data transmission besides a Bluetooth module, such as a component capable of wired or wireless transmission, and the present invention is not limited thereto.

[0073] Preferably, the head posture information includes horizontal head orientation information and / or vertical head orientation information, and the controller 120 is configured to determine whether the horizontal head orientation information satisfies a first preset condition and / or whether the vertical head orientation information satisfies a second preset condition. If the horizontal head orientation information does not satisfy the first preset condition and / or the vertical head orientation information does not satisfy the second preset condition, the controller 120 instructs the issuance of an alarm signal and / or activation of a protection mechanism to prevent the subject from executing the next operation when not facing the display screen 130. Thus, in some embodiments, by determining whether the subject's horizontal head orientation information satisfies the first preset condition, it is possible to determine whether the subject's head is facing the display screen 130 in the left-right direction (horizontally, i.e., parallel to the ground), thereby determining whether the subject's line of sight is facing the display screen. In other embodiments, by determining whether the subject's vertical orientation information satisfies the second preset condition, it is possible to determine whether the subject's head is facing the display screen 130 in the up-down direction (vertically, i.e., perpendicular to the ground), thereby determining whether the subject's line of sight is facing the display screen. In order to further improve the accuracy of line of sight detection, preferably, the controller 120 determines that the subject's line of sight is facing the display screen 130 only when the horizontal orientation information of the subject's head meets the first preset condition and the vertical orientation information of the head meets the second preset condition, that is, when the subject's head faces the display screen 130 in both the horizontal and vertical directions, so that line of sight detection can be completed more accurately.

[0074] Preferably, the head-mounted device includes at least one first magnetic sensor 111 in communication with the controller 120, the first magnetic sensor 111 being used to detect the angle between the horizontal orientation of the subject's head and a predetermined magnetic field to obtain horizontal head orientation information, and / or being used to detect the angle between the vertical orientation of the subject's head and the predetermined magnetic field to obtain vertical head orientation information. The first magnetic sensor 111 can be any magnetic sensor in the prior art, such as a geomagnetic sensor, and the predetermined magnetic field can be a geomagnetic field or a pre-set fixed magnetic field.

[0075] The following uses the geomagnetic sensor as an example to illustrate the measurement principle of the head posture information in the present invention. Please refer to Figure 2 and Figure 3 ,in Figure 2 The following schematically shows the principle of measuring the horizontal head orientation information provided by one embodiment of the present invention. Figure 3 The following schematically shows the principle of measuring the vertical orientation information of the head provided by one embodiment of the present invention. Figure 2 and Figure 3As shown, direction N represents the direction of the geomagnetic field, direction S represents the vertical orientation of the subject's head, the first magnetic sensor 111 is a geomagnetic sensor and preferably a three-axis geomagnetic sensor, the predetermined magnetic field is the geomagnetic field, and the first magnetic sensor 111 is used to detect the angle α between the horizontal orientation of the subject's head and the geomagnetic field to obtain the horizontal orientation information of the head, and to detect the angle θ between the vertical orientation of the subject's head and the geomagnetic field to obtain the vertical orientation information of the head. Since the geomagnetic field exists naturally, by adopting a geomagnetic sensor as the first magnetic sensor 111 in the present invention, there is no need to set up an additional magnetic field, which greatly simplifies the overall structure of the line of sight detection device provided by the present invention. The first magnetic sensor 111 can conveniently and quickly measure the angle α between the horizontal orientation of the subject's head and the geomagnetic field, that is, the subject's head orientation information, and the angle θ between the vertical orientation of the subject's head and the geomagnetic field, that is, the subject's head vertical orientation information.

[0076] Preferably, in order to improve the measurement accuracy, the head mounted device 110 includes a plurality of (two or more) first magnetic sensors 111, such as Figure 2 As shown, in this embodiment, the head-mounted device 110 includes three first magnetic sensors 111, and the three first magnetic sensors 111 are located at different parts of the head-mounted device 110. For example, when the head-mounted device 110 is a pair of glasses, the three first magnetic sensors 111 are respectively arranged at different parts of the glasses frame. Preferably, two of the first magnetic sensors 111 are located on the same side bracket of the glasses frame, and the other first magnetic sensor 111 is located on the other side bracket of the glasses frame. Thus, three measurement results can be obtained based on the three first magnetic sensors 111, and by comparing the three measurement results, erroneous measurement results can be effectively eliminated.

[0077] Preferably, the display screen 130 is provided with at least one second magnetic sensor 131 in communication with the controller 120. The second magnetic sensor 131 can be installed at any position on the display screen 130. The second magnetic sensor 131 is used to detect the angle between the screen direction of the display screen 130 and the predetermined magnetic field. The controller 120 is used to determine whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen 130 and the predetermined magnetic field is within a first preset range. If not, it is determined that the subject is not facing the display screen 130. The controller 120 instructs to issue an alarm signal and / or activate a protection mechanism to prevent the subject from performing the next operation when not facing the display screen 130. Therefore, the present invention uses whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen 130 and the predetermined magnetic field is within the first preset range as one of the bases for determining whether the subject is facing the display screen 130. That is, the present invention takes into account the influence of the posture of the display screen 130 on the line of sight detection, which can further improve the accuracy of line of sight detection. In order to improve the measurement accuracy and prevent erroneous measurement results from affecting the final result of the line of sight detection, at least two second magnetic sensors 131 may be provided on the display screen 130 .

[0078] The following uses the geomagnetic sensor as an example to explain the measurement principle of the orientation information of the display screen 130 in the present invention. Please refer to Figure 2 and Figure 4 ,in Figure 4 The main view of the display screen 130 is schematically shown. Figure 2 and Figure 4 As shown, direction G represents the direction of gravity, the screen direction is parallel to the gravity direction G, and the geomagnetic field direction N is perpendicular to the gravity direction G. The second magnetic sensor 131 is a geomagnetic sensor and preferably a three-axis geomagnetic sensor. The second magnetic sensor 131 is used to detect the angle β between the screen direction of the display screen 130 and the geomagnetic field. The controller 120 is used to determine whether the absolute value of the difference between the angle α between the horizontal direction of the subject's head and the geomagnetic field and the angle β between the screen direction of the display screen 130 and the geomagnetic field is within a first preset range. If not, it is determined that the subject is not facing the display screen 130. It should be noted that this embodiment uses a geomagnetic sensor as an example of the magnetic sensor in the present invention. As those skilled in the art will understand, the first magnetic sensor 111 and the second magnetic sensor 131 can also be other magnetic sensors besides geomagnetic sensors, and the predetermined magnetic field can also be other fixed magnetic fields besides the geomagnetic field. The present invention is not limited to this.

[0079] In order to facilitate data transmission, the display screen 130 may be provided with a second data transceiver 132 , so that data can be transmitted between the display screen 130 and the controller 120 via the second data transceiver 132 .

[0080] Preferably, the controller 120 is configured to determine whether the angle between the vertical orientation of the subject's head and the predetermined magnetic field (e.g., the Earth's magnetic field) is within a second preset range. If not, the controller 120 issues an alarm signal and / or activates a protection mechanism. Thus, when the angle between the vertical orientation of the subject's head and the predetermined magnetic field (e.g., the Earth's magnetic field) exceeds the second preset range, it is determined that the subject is not facing the display screen 130, and the controller 120 issues an alarm signal and / or activates a protection mechanism to prevent the subject from executing the next operation when not facing the display screen 130.

[0081] Preferably, the head-mounted device 110 is also used to detect the subject's head movement information and transmit the head movement information to the controller 120; the controller 120 is used to determine whether the head movement information meets a third preset condition. If not, the controller 120 instructs to issue an alarm signal and / or activate a protection mechanism to prevent the subject from performing the next operation when not facing the display screen 130. Because after the subject's line of sight is determined to be facing the display screen 130 based on the subject's head posture information and the preset conditions, the subject may move his head so that his line of sight is no longer facing the display screen 130. Therefore, the present invention can determine whether the subject's head is beyond the movable range by detecting the subject's head movement information and determining whether the head movement information meets the third preset condition, and then can issue an alarm signal and / or activate a protection mechanism when the subject's head is beyond the movable range to prevent misoperation.

[0082] For preference, please refer to Figure 5 and Figure 6 ,in Figure 5 The following schematically shows an installation diagram of the acceleration sensor 112 provided in one embodiment of the present invention on the head mounted device 110. Figure 6 The measurement principle diagram of the acceleration sensor 112 provided in one embodiment of the present invention is schematically shown. Figure 5 and Figure 6As shown, direction D represents the acceleration direction. The head-mounted device 110 also includes an acceleration sensor 112 connected to the controller 120. The acceleration sensor 112 can be installed at any position of the head-mounted device 110. The acceleration sensor 112 is used to detect the acceleration of the subject's head movement. The controller 120 is also used to integrate the head movement acceleration to obtain the subject's head movement distance, and to determine whether the head movement distance is within a third preset range. If not, the controller 120 sends an alarm signal and / or activates a protection mechanism. In this embodiment, the acceleration sensor 112 is preferably a three-axis acceleration sensor 112, which can measure the acceleration information of the subject's head in the X, Y, and Z directions. like Figure 6 As shown, O(A, B, C) is the position of the acceleration sensor 112 in a normal posture (i.e., the subject's line of sight faces the display screen 130) (i.e., the zero point position), and O'(A', B', C') is the position of the acceleration sensor 112 at time T. represents the head movement vector of the subject within time T, where:

[0083]

[0084]

[0085]

[0086] Therefore, the acceleration sensor 112 can measure the movement distance of the subject's head in different directions, and then determine whether the movement distance of the subject's head exceeds the movable distance (the third preset range). When the subject's head exceeds the movable range, an alarm signal can be issued and / or a protection mechanism can be activated to prevent misoperation.

[0087] To ensure that the controller 120 can correctly determine the zero point position of the accelerometer 112 and determine the origin position of the coordinate system, calibration is required in advance. During calibration, the subject maintains a normal operating posture (i.e., facing the display screen) and initiates the calibration process by performing a certain action. For example, for a surgical robot system, the subject (i.e., the doctor) can start the calibration process by pinching the opening and closing joints of the main control arm or other methods. The controller 120 establishes a coordinate system with the current position of the accelerometer 112 as the origin and integrates the acceleration to determine the distance the subject's head has moved in real time.

[0088] Preferably, please refer to Figure 7 and Figure 8 ,in Figure 7A schematic diagram of installing the gyroscope 113 provided in one embodiment of the present invention on the head mounted device 110 is shown schematically. Figure 8 The measurement principle diagram of the gyroscope 113 provided in one embodiment of the present invention is schematically shown. Figure 7 and Figure 8 As shown, direction G represents the direction of gravity. The head-mounted device 110 further includes a gyroscope 113 connected to the controller 120. The gyroscope 113 can be installed at any position of the head-mounted device 110. The gyroscope 113 is used to detect the vertical posture vector of the head of the subject. The controller 120 is also used to determine whether the vertical posture vector of the head is within a fourth preset range. If not, the controller 120 instructs to issue an alarm signal and / or activate a protection mechanism to prevent the subject from performing the next operation when not facing the display screen 130. Figure 7 As shown, in a normal posture (i.e., the subject's line of sight faces the display screen 130), the direction of the subject's head posture vector is consistent with the gravity direction G, as shown in FIG. Figure 8 As shown, is the head posture vector of the subject in normal posture, and is the maximum movable posture vector (i.e., the upper and lower limits of the fourth preset range), where Indicates the maximum movable posture vector when the subject raises his head, and The angle between them can be 45°. Indicates the maximum movable posture vector of the subject when lowering his head, and The angle between them can be 45°. Indicates the boundary of the maximum movable visual field when the subject looks up. Indicates the boundary of the maximum movable visual field range when the subject lowers his head. If the vertical posture vector of the subject's head measured by the gyroscope 113 is within the fourth preset range (i.e., not exceeding and ), it means that the person raising or lowering his head does not exceed the normal field of vision (i.e., does not exceed and ), thereby, the vertical posture vector of the subject's head can be measured by the gyroscope 113, and then it can be determined whether the vertical posture vector of the subject's head exceeds the movable range (i.e., the fourth preset range), so that when the subject's field of view exceeds the normal field of view, an alarm signal can be issued and / or a protection mechanism can be activated to prevent misoperation.

[0089] The above-mentioned sight line detection device can be used in a variety of scenarios for identifying whether the sight line is within a predetermined range. The following will illustrate its application in a surgical robot as an example.

[0090] The present invention also provides a surgical robot system, the surgical robot system comprising the sight line detection device as described above. Figures 9 to 11 ,in, Figure 9 The overall structural diagram of a surgical robot system provided by one embodiment of the present invention is schematically shown; Figure 10 The following schematically shows the structure of a robotic arm of an operating table provided in one embodiment of the present invention. Figure 11 The following schematically shows the structure of the doctor console provided by one embodiment of the present invention. Figures 9 to 11 As shown, the surgical robot system includes a control end and an execution end. The control end includes a doctor's console 10 equipped with a master control arm 11. The execution end includes equipment such as an operating table 20 and a side trolley 30. The patient lies on the operating table 20 for surgery. The side trolley 30 is equipped with a robotic arm 31 for mounting surgical instruments and an endoscope. The robotic arm 31, surgical instruments, and endoscope have a predetermined mapping relationship with the master control arm 11, forming a master-slave relationship. The robotic arm 31, surgical instruments, and endoscope move in various directions based on the movement of the master control arm 11 to complete the surgery. The doctor's console 10 includes a display screen 130, a multi-axis robotic arm (i.e., the main control arm 11) and the line of sight detection device as described above. The doctor (the subject) remotely controls the robotic arm 31 of the operating trolley 20 to perform surgery by operating the multi-axis robotic arm. The display screen 130 can display the intraoperative process in the abdominal cavity obtained from the endoscope when the robotic arm 31 performs surgery. The side trolley 30 includes two or more robotic arms 31. The doctor controls two of the robotic arms 31 through the multi-axis robotic arm (i.e., the main control arm 11) of the doctor's console 10. The actions of the two robotic arms 31 operating surgical instruments (such as clamping and resection of lesions, etc.) can be captured by the endoscope camera and displayed on the display screen 130 of the doctor's console 10.

[0091] When performing line of sight detection, the doctor wears the head-mounted device 110 and maintains a normal operating posture. The medical control system is enabled, the head-mounted device 110 detects the doctor's head posture information and transmits the head posture information to the controller 120. The controller 120 determines whether the doctor's line of sight is facing the display screen 130 based on the head posture information and preset conditions. If the judgment result is that the doctor's line of sight is facing the display screen 130, a master-slave relationship is established between the master control arm 11 and the robotic arm 31, and the master-slave connection state is entered. Then, the doctor can operate the master control arm 11 normally and control the robotic arm 31 to perform surgery. If the judgment result is that the doctor's line of sight is not facing the display screen 130, the system automatically activates the protection mechanism and / or issues an alarm signal. The master control arm 11 cannot operate the robotic arm 31 to perform surgery and enters a locked state of the robotic arm 31. The doctor needs to lightly touch the screen unlock button or pinch the end pinch joint of the master control arm 11 at the same time or perform other unlocking actions, re-perform line of sight detection, and after the judgment result as described above is satisfied that the doctor's line of sight is facing the display screen 130, the robotic arm 31 is unlocked, and the doctor can perform surgery normally. Therefore, by adopting the surgical robot system provided by the present invention, the doctor's line of sight detection can be successfully completed, ensuring that the master-slave control relationship between the main control arm 11 and the robotic arm 31 can only be controlled when the doctor's line of sight is facing the display screen 130. Otherwise, the robotic arm 31 will be automatically locked, thereby effectively avoiding misoperation and improving safety performance during the operation.

[0092] Specifically, when performing line of sight detection, the doctor wears the head-mounted device 110 and maintains a normal operating posture. The first magnetic sensor 111 located on the head-mounted device 110 detects the angle α between the horizontal orientation of the doctor's head and a predetermined magnetic field, such as the geomagnetic field, and / or the angle θ between the vertical orientation of the doctor's head and a predetermined magnetic field, such as the geomagnetic field, and transmits the measured angle α and angle θ to the controller 120. The second magnetic sensor 131 located on the display screen 130 detects the angle β between the screen direction of the display screen 130 and a predetermined magnetic field, such as the geomagnetic field, and transmits the measured angle β to the controller 120. The controller 120 determines whether the absolute value of the difference between the angle α and the angle β, that is, |α-β|, is within a first preset range, and / or determines whether the angle The doctor then checks whether the degree θ is within a second preset range. If |α-β| is within the first preset range and / or θ is within the second preset range, the doctor's line of sight is facing the display screen 130, and the doctor can operate the master control arm 11 normally. If |α-β| exceeds the first preset range and / or θ exceeds the second preset range, the doctor's line of sight is not facing the display screen 130. At this time, the controller 120 automatically activates a protection mechanism and / or issues an alarm signal, and the doctor cannot operate the master control arm 11. The robotic arm 31 is locked. The doctor needs to touch the screen unlock button, pinch the end of the master control arm 11, or perform other unlocking actions to re-test the line of sight. If the above judgment result is satisfied that the doctor's line of sight is facing the display screen 130, the robotic arm 31 is unlocked. The first preset range and the second preset range can be set according to specific circumstances. For example, the first preset range is (70°, 110°).

[0093] When the sight line detection device detects that the doctor's sight line is facing the display screen 130, that is, when the doctor is observing the display screen, the doctor maintains a normal operating posture, the medical control system is enabled, and the doctor confirms the operating posture by pinching the main control arm 11 to open and close the joint or by other means. In order to further accurately limit the doctor's sight line to a predetermined range and ensure that the doctor obtains more accurate image information, for example, the most accurate image information is obtained when the doctor's sight line is perpendicular to the central axis of the display screen, the controller 120 confirms the zero point position of the acceleration sensor 112 and establishes a three-dimensional coordinate system. The acceleration sensor 112 located on the head-mounted device 110 detects the acceleration of the doctor's head movement (that is, the acceleration sensor 112) and transmits the measured acceleration information to the controller 120. The controller 120 integrates the acceleration based on the zero point to calculate the distance the doctor's head has moved within the three-dimensional coordinate system and determines whether the movement distance is within a third preset range. If the doctor's head movement distance exceeds the third preset range, the controller 120 issues an alarm signal and / or activates a protection mechanism, controlling the robotic arm 31 to lock. At this time, the doctor needs to lightly press the screen unlock button or simultaneously pinch the end of the master control arm 11 to re-test the line of sight. If the doctor's head movement distance is within the third preset range, the doctor can operate the master control arm 11 normally. Thus, the acceleration sensor 112 can measure the doctor's head movement distance in real time. If the doctor's line of sight again deviates from the display screen 130 due to left and right head movement, the controller 120 issues an alarm signal and / or activates a protection mechanism, thereby preventing the doctor from operating the master control arm 11 when the line of sight is not facing the display screen 130.

[0094] In another or further embodiment, in order to more accurately limit the doctor's line of sight and ensure that the doctor obtains accurate image information, when the line of sight detection device detects that the doctor's line of sight is facing the display screen 130, the doctor maintains a normal operating posture, the medical control system is enabled, and the gyroscope 113 located on the head-mounted device 110 detects the doctor's head vertical posture vector and transmits the head vertical posture vector to the controller 120. The controller 120 determines whether the head vertical posture vector is within a fourth preset range. If the doctor's head vertical posture vector exceeds the fourth preset range, the controller 120 activates the protection mechanism and / or issues an alarm signal to control the robotic arm 31 to be in a locked state. At this time, the doctor needs to touch the screen unlock button, pinch the end pinch joint of the main control arm 11, or perform other unlocking actions to re-perform line of sight detection; if the doctor's head vertical posture vector is within the fourth preset range, the doctor can operate the main control arm 11 normally. Therefore, the gyroscope 113 can measure the vertical posture vector of the doctor's head in real time, and then when the doctor's line of sight deviates from the display screen 130 again due to raising or lowering his head, the protection mechanism can be activated, thereby preventing the doctor from operating the main control arm 11 when his line of sight is not facing the display screen 130.

[0095] Corresponding to the above-mentioned surgical robot system, the present invention also provides a control method for the surgical robot system, please refer to Figure 12 , which schematically shows a flow chart of a control method for a surgical robot system provided by an embodiment of the present invention, such as Figure 12 As shown, the control method of the surgical robot system includes the following steps:

[0096] Step S1: obtaining the head posture information of the subject;

[0097] Step S2: determining whether the head posture information meets a preset condition;

[0098] If yes, then execute the following step S3:

[0099] Step S3, determining that the subject's line of sight is facing the display screen;

[0100] If not, then execute the following step S4:

[0101] Step S4: issuing an alarm signal and / or controlling the robotic arm to be in a locked state.

[0102] Thus, the present invention obtains the head posture information of the doctor (the subject) and determines whether the head posture information meets the preset conditions, and then determines whether the doctor is facing the display screen. If the judgment result is that the doctor's line of sight is facing the display screen, a master-slave relationship is established between the master control arm and the robotic arm, and a master-slave connection state is entered, and then the doctor can normally operate the master control arm and control the robotic arm to perform the operation; if the judgment result is that the doctor's line of sight is not facing the display screen, the system automatically sends an alarm signal and / or activates a protection mechanism, and the master control arm cannot operate the robotic arm to perform the operation, and enters a robotic arm locked state. The doctor needs to touch the screen unlock button, pinch the end of the master control arm to pinch the joint, or perform other unlocking actions, re-perform the line of sight detection, and after the judgment result as described above is satisfied that the doctor's line of sight is facing the display screen, the robotic arm is unlocked, and the doctor can perform the operation normally. Thus, the present invention can successfully complete the doctor's line of sight detection, ensuring that the master control arm and the robotic arm can only be controlled when the doctor's line of sight is facing the display screen, otherwise the robotic arm will be automatically locked, thereby effectively avoiding misoperation and improving the safety performance during the operation.

[0103] Preferably, the step of obtaining the head posture information of the subject includes:

[0104] Obtaining horizontal head orientation information and vertical head orientation information of the subject;

[0105] The determining whether the head posture information meets a preset condition includes:

[0106] It is determined whether the horizontal head orientation information satisfies a first preset condition and / or it is determined whether the vertical head orientation information satisfies a second preset condition.

[0107] Preferably, the obtaining of the subject's head horizontal orientation information and / or head vertical orientation information includes:

[0108] The angle between the horizontal orientation of the subject's head and the predetermined magnetic field and / or the angle between the vertical orientation of the subject's head and the predetermined magnetic field is obtained.

[0109] Preferably, the control method further includes:

[0110] Acquiring an angle between a screen direction of the display screen and the predetermined magnetic field;

[0111] The determining whether the head horizontal orientation information satisfies a first preset condition includes:

[0112] It is determined whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range.

[0113] Preferably, the determining whether the head vertical orientation information satisfies a second preset condition includes:

[0114] It is determined whether the angle between the vertical direction of the subject's head and the predetermined magnetic field is within a second preset range.

[0115] Preferably, the control method includes:

[0116] Obtaining the subject's head movement information;

[0117] Determining whether the head movement information meets a third preset condition;

[0118] If not, an alarm signal is issued and / or the robotic arm is controlled to be in a locked state.

[0119] Preferably, obtaining the subject's head movement information includes:

[0120] Obtain the head movement distance of the subject;

[0121] The determining whether the head movement information satisfies a third preset condition includes:

[0122] Determine whether the head movement distance is within a third preset range.

[0123] Preferably, obtaining the head movement distance of the subject includes:

[0124] Obtain the subject's head movement acceleration;

[0125] The head movement acceleration is integrated to obtain the head movement distance of the subject.

[0126] Preferably, obtaining the subject's head movement information includes:

[0127] Obtain the vertical posture vector of the subject's head;

[0128] The determining whether the head movement information satisfies a third preset condition includes:

[0129] Determine whether the head vertical posture vector is within a fourth preset range.

[0130] Based on the above similar inventive concepts, the present invention also provides a readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the control method of the surgical robot system described above can be implemented. The storage medium provided by the present invention obtains the head posture information of the subject and determines whether the head posture information meets the preset conditions, thereby determining whether the subject is facing the display screen. Therefore, compared with the method of realizing line of sight detection by collecting eye movement data, the present invention has lower requirements on the position stability of the doctor (subject) during the line of sight detection process, and will not fail to complete the line of sight detection because the doctor deviates from the predetermined test range for some reason, thereby greatly improving the operability of the line of sight detection. In addition, when the present invention determines that the doctor's line of sight is not facing the display screen, it will automatically send an alarm signal and / or activate a protection mechanism to put the robotic arm used to perform the operation in a locked state, thereby effectively preventing the doctor from operating the robotic arm when his line of sight is not facing the display screen, further improving the safety during the operation and effectively preventing misoperation.

[0131] The readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0132] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0134] It should be noted that the devices and methods disclosed in the embodiments of this document may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the devices, methods, and computer program products according to the various embodiments of this document. In this regard, each box in the flowchart or block diagram may represent a module, program, or portion of code, wherein the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function, and the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0136] In summary, compared with the prior art, the line of sight detection device, surgical robot system, control method and storage medium provided by the present invention have the following advantages: the present invention obtains the head posture information of the subject and determines whether the head posture information meets the preset conditions, and then determines whether the subject is facing the display screen. Therefore, compared with the method of realizing line of sight detection by collecting eye movement data, the present invention has lower requirements on the position stability of the subject during the line of sight detection process, and will not fail to complete the line of sight detection because the subject deviates from the predetermined test range for some reason, thereby greatly improving the operability of the line of sight detection. In addition, when the present invention determines that the doctor's (subject's) line of sight is not facing the display screen, it will automatically send an alarm signal and / or activate a protection mechanism to put the robotic arm used to perform the operation in a locked state, thereby effectively preventing the doctor from operating the robotic arm when his line of sight is not facing the display screen, further improving the safety during the operation and effectively preventing misoperation.

[0137] The above description is merely a description of preferred embodiments of the present invention and does not limit the scope of the invention. Any changes or modifications made by persons skilled in the art based on the above disclosure are intended to be protected by the claims. Obviously, various modifications and variations may be made by persons skilled in the art without departing from the spirit and scope of the present invention. Thus, to the extent such modifications and variations fall within the scope of the claims and their equivalents, the present invention is intended to encompass such modifications and variations.

Claims

1. A sight line detection device, characterized in that: A head-mounted device and controller including communication connections; The head-mounted device is used to detect head posture information of the subject and transmit the head posture information to the controller; The controller is used to determine whether the subject's line of sight is facing the display screen according to the head posture information and preset conditions; The head posture information includes head horizontal orientation information; The head-mounted device includes at least one first magnetic sensor in communication with the controller, the first magnetic sensor being configured to detect an angle between a horizontal orientation of the subject's head and a predetermined magnetic field to obtain horizontal orientation information of the head; The display screen is provided with at least one second magnetic sensor in communication with the controller, the second magnetic sensor being used to detect an angle between a screen direction of the display screen and a predetermined magnetic field; The controller is used to determine whether the absolute value of the difference between the angle between the horizontal direction of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range. If not, it is determined that the subject's line of sight is not facing the display screen, and an alarm signal is issued and / or a protection mechanism is activated.

2. The sight line detection device according to claim 1, characterized in that: The head posture information also includes head vertical orientation information. The controller is also used to determine whether the head vertical orientation information meets a second preset condition. If the head vertical orientation information does not meet the second preset condition, the controller instructs to issue an alarm signal and / or start a protection mechanism.

3. The sight line detection device according to claim 2, characterized in that: The first magnetic sensor is further configured to detect an angle between the vertical orientation of the subject's head and the predetermined magnetic field to obtain the vertical orientation information of the head.

4. The sight line detection device according to claim 3, characterized in that: The controller is further configured to determine whether the angle between the vertical orientation of the subject's head and the predetermined magnetic field is within a second preset range; if not, an alarm signal is issued and / or a protection mechanism is activated.

5. The sight line detection device according to claim 1, wherein: The head-mounted device is further used to detect head movement information of the subject and transmit the head movement information to the controller; The controller is used to determine whether the head movement information meets a third preset condition, and if not, instruct to send an alarm signal and / or start a protection mechanism.

6. The sight line detection device according to claim 5, characterized in that: The head-mounted device includes an acceleration sensor in communication with the controller, and the acceleration sensor is used to detect the acceleration of the head movement of the subject; The controller is used to integrate the head movement acceleration to obtain the head movement distance of the subject, and to determine whether the head movement distance is within a third preset range. If not, an alarm signal is issued and / or a protection mechanism is activated.

7. The sight line detection device according to claim 5, characterized in that: The head-mounted device includes a gyroscope in communication with the controller, and the gyroscope is used to detect the vertical posture vector of the head of the subject; The controller is further configured to determine whether the head vertical posture vector is within a fourth preset range, and if not, to indicate that an alarm signal is issued and / or a protection mechanism is activated.

8. A surgical robot system, characterized in that: It includes a display screen, a robotic arm and a line of sight detection device as described in any one of claims 1 to 7, the display screen is communicatively connected to the controller, the display screen is provided with at least one second magnetic sensor communicatively connected to the controller, the second magnetic sensor is used to detect the angle between the screen direction of the display screen and a predetermined magnetic field, the controller is used to: determine whether the absolute value of the difference between the angle between the horizontal direction of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range; if not, determine that the subject's line of sight is not facing the display screen, and control the surgical robot system to issue an alarm signal and / or put the robotic arm into a locked state.

9. A control method for a surgical robot system, characterized in that: The surgical robot system includes a display screen and a robotic arm, and the control method includes: Obtaining the subject's head posture information; Determining whether the head posture information meets a preset condition; If yes, it is determined that the subject's line of sight is facing the display screen; If not, an alarm signal is issued and / or the robotic arm is controlled to be in a locked state; The step of obtaining the head posture information of the subject includes: Obtaining the horizontal orientation information of the subject's head; The determining whether the head posture information meets a preset condition includes: Determining whether the head horizontal orientation information meets a first preset condition; The obtaining of the horizontal orientation information of the subject's head includes: Obtaining the angle between the horizontal orientation of the subject's head and the predetermined magnetic field; The control method further includes: Acquiring an angle between a screen direction of the display screen and the predetermined magnetic field; The determining whether the head horizontal orientation information satisfies a first preset condition includes: It is determined whether the absolute value of the difference between the angle between the horizontal orientation of the subject's head and the predetermined magnetic field and the angle between the screen direction of the display screen and the predetermined magnetic field is within a first preset range.

10. The control method of the surgical robot system according to claim 9, characterized in that: The step of obtaining the head posture information of the subject further includes: Obtaining the vertical orientation information of the subject's head; The determining whether the head posture information meets a preset condition further includes: It is determined whether the head vertical orientation information meets a second preset condition.

11. The control method of the surgical robot system according to claim 10, characterized in that: The obtaining of the vertical orientation information of the subject's head includes: The angle between the vertical direction of the subject's head and the predetermined magnetic field is obtained.

12. The control method of the surgical robot system according to claim 11, characterized in that: The determining whether the head vertical orientation information satisfies a second preset condition includes: It is determined whether the angle between the vertical direction of the subject's head and the predetermined magnetic field is within a second preset range.

13. The control method of the surgical robot system according to claim 9, characterized in that: The control method includes: Obtaining the subject's head movement information; Determining whether the head movement information meets a third preset condition; If not, an alarm signal is issued and / or the robotic arm is controlled to be in a locked state.

14. The control method of the surgical robot system according to claim 13, characterized in that: The step of obtaining the head movement information of the subject includes: Obtain the head movement distance of the subject; The determining whether the head movement information satisfies a third preset condition includes: Determine whether the head movement distance is within a third preset range.

15. The control method of the surgical robot system according to claim 14, characterized in that: The step of obtaining the head movement distance of the subject includes: Obtain the subject's head movement acceleration; The head movement acceleration is integrated to obtain the head movement distance of the subject.

16. The control method of the surgical robot system according to claim 14, characterized in that: The step of obtaining the head movement information of the subject includes: Obtain the vertical posture vector of the subject's head; The determining whether the head movement information satisfies a third preset condition includes: Determine whether the head vertical posture vector is within a fourth preset range.

17. A readable storage medium, characterized in that The readable storage medium stores a computer program, and when the computer program is executed by the processor, the control method of the surgical robot system according to any one of claims 9 to 16 is implemented.

Images