Methods for detecting the relative relationship between the line of sight and the target object and surgical robot systems

Abstract

This invention provides a method, a surgical robot system, a control method, and a storage medium for detecting the relative relationship between a gaze and a target object. The method includes: acquiring the relative positional relationship between the subject's face and the target object; acquiring the subject's gaze orientation range based on the relative positional relationship; determining whether the subject's gaze orientation range at least partially covers the target object; and if not, issuing a warning message and / or issuing a state change command. Compared to methods that detect gaze by collecting eye movement data, this invention has lower requirements for the positional stability of the subject during gaze detection. It prevents gaze detection from failing due to the subject deviating from the predetermined testing range for any reason, greatly improving the operability and accuracy of gaze detection.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a method for detecting the relative relationship between a line of sight and a target object, a surgical robot system, a control method, and a storage medium. Background Technology

[0002] Current medical devices typically track a doctor's gaze by collecting eye movement data. Common eye trackers use infrared illumination to project light onto the doctor's eyes, creating a light spot on the cornea. This light spot, along with images of the doctor's pupils, is then used to calculate the doctor's gaze direction. However, ambient light and other interference factors can significantly disrupt the gaze detection process. Furthermore, the limited field of view of the imaging devices on eye trackers requires a certain level of stability in the doctor's position during the detection process. If the doctor deviates from the predetermined testing range for any reason, the gaze detection cannot be successfully completed, and the system will struggle to determine whether the doctor is looking at the display. Such misoperation by the doctor could potentially harm the patient. Currently, no effective solution has been proposed to address the problem of the inability of existing technologies to accurately determine a doctor's gaze. Summary of the Invention

[0003] The purpose of this invention is to provide a method, a surgical robot system, a control method, and a storage medium for detecting the relative relationship between the line of sight and the target object, which can effectively determine whether the doctor is looking at the display device to ensure the safety of the surgery.

[0004] To solve the above-mentioned technical problems, the present invention provides a method for detecting the relative relationship between a line of sight and a target object, comprising:

[0005] Obtain the relative positional relationship between the subject's face and the target object;

[0006] Based on the relative positional relationship between the subject's face and the target object, the subject's gaze orientation range is obtained;

[0007] Determine whether the range of the subject's line of sight at least partially covers the target object;

[0008] If not, issue a warning message and / or issue a status change command.

[0009] Optionally, the subject wears a head-mounted device, and acquiring the relative positional relationship between the subject's face and the target object includes:

[0010] Obtain the relative positional relationship between the head-mounted device and the target object;

[0011] The relative positional relationship between the subject's face and the target object is obtained based on the relative positional relationship between the head-mounted device and the target object, and the relative positional relationship between the subject's face and the head-mounted device.

[0012] Optionally, the head-mounted device is equipped with multiple magnetic sensors, and the target object is equipped with a magnetic field generator;

[0013] The step of obtaining the relative positional relationship between the head-mounted device and the target object includes:

[0014] The three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system generated by the magnetic field generator are obtained respectively.

[0015] The relative positional relationship between the head-mounted device and the target object is obtained based on the three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system and the mapping relationship between the magnetic field coordinate system and the target object coordinate system.

[0016] Optionally, obtaining the three-dimensional coordinates of the plurality of magnetic sensors in the magnetic field coordinate system includes:

[0017] Based on the pre-obtained position calibration results, the three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system are obtained respectively.

[0018] Optionally, the target object is a display device, and the step of determining whether the subject's gaze orientation range at least partially covers the target object includes:

[0019] Determine whether the range of the subject's gaze direction at least partially overlaps with the target object.

[0020] To achieve the above objectives, the present invention also provides a control method for a surgical robot system, the surgical robot system including a display device and a robotic arm, the control method comprising:

[0021] Using the method described above for detecting the relative relationship between the line of sight and the target object, it is determined whether the subject's line of sight is facing the display device; and

[0022] If it is determined that the subject's gaze is not directed toward the display device, a warning message is issued and / or the robotic arm is locked.

[0023] To achieve the above objectives, the present invention also provides a surgical robot system, including a display device, a controller, and a robotic arm. The display device is communicatively connected to the controller, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the method for detecting the relative relationship between the line of sight and the target object or the control method of the surgical robot system described above.

[0024] Optionally, the surgical robot system includes a head-mounted device that is communicatively connected to the controller. The head-mounted device includes an eyeglass frame with multiple magnetic sensors mounted on its support. The display device is equipped with a magnetic field generator that generates a magnetic field within a predetermined range. The magnetic sensors induce a corresponding current in the magnetic field. The controller calculates the relative positional relationship between the head-mounted device and the display device based on the induced current.

[0025] Optionally, at least two of the plurality of magnetic sensors are located on supports on different sides of the eyeglass frame.

[0026] Optionally, the surgical robot system includes a doctor's console and a side trolley. The doctor's console includes the display device, the controller, and the main control arm. The side trolley includes the robotic arm. The controller is used to determine if the subject's gaze is directed towards the display device, and then establish a master-slave relationship between the main control arm and the robotic arm.

[0027] To achieve the above objectives, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method for detecting the relative relationship between the line of sight and the target object or the control method for a surgical robot system described above.

[0028] Compared with existing technologies, the method, surgical robot system, control method, and storage medium for detecting the relative relationship between the gaze and the target object provided by this invention have the following advantages: This invention first acquires the relative positional relationship between the subject's face and the target object (e.g., a display device); then, based on this relative positional relationship, it acquires the subject's gaze orientation range; finally, it determines whether the subject's gaze orientation range at least partially covers the target object (e.g., the display device), i.e., whether the subject's gaze orientation range at least partially overlaps with the target object. If so, it is determined that the subject's gaze is facing the target object (e.g., the display device); otherwise, a warning message and / or a state change command are issued. Therefore, compared with methods that achieve gaze detection by collecting eye movement data, this invention has lower requirements for the positional stability of the subject during the gaze detection process. It will not fail to complete the gaze detection because the subject deviates from the predetermined test range for some reason, greatly improving the operability and accuracy of gaze detection. Furthermore, when the present invention determines that the subject's gaze is not directed towards the target object, it automatically issues a warning message and / or a status change command, thereby effectively preventing misoperation and improving safety during operation. For example, in a surgical robot system, when the present invention determines that the doctor's (subject's) gaze is not directed towards the display device (target object), it automatically issues a warning message and / or activates a protection mechanism to lock the robotic arm used to perform the surgery. This effectively prevents the doctor from operating the robotic arm when their gaze is not directed towards the display device, further improving safety during surgery and effectively preventing misoperation. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a method for detecting the relative relationship between a line of sight and a target object according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the subject's line of sight facing the target object in one embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram showing the relative positional relationship between the target object and the head-mounted device in the X / Y directions according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram showing the relative positional relationship between the target object and the head-mounted device in the X / Z directions according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram illustrating the working principle of a magnetic field generator according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of a magnetic field coordinate system in one embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram illustrating the working principle of a magnetic sensor in one embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the coordinate measurement principle in one embodiment of the present invention;

[0037] Figure 9 This is a schematic diagram of the coordinate measurement results of a magnetic sensor in one embodiment of the present invention;

[0038] Figure 10 This is a schematic diagram of the structure of a surgical robot system according to an embodiment of the present invention;

[0039] Figure 11 This is a schematic diagram of the structure of a doctor's console in one embodiment of the present invention;

[0040] Figure 12 This is a block diagram of the controller in one embodiment of the present invention;

[0041] Figure 13 This is a schematic diagram of the coordinate system of a display device according to an embodiment of the present invention;

[0042] Figure 14 This is a schematic diagram showing the relative positional relationship between the magnetic field generator and the display device of the doctor's console in one embodiment of the present invention;

[0043] Figure 15 This is a schematic diagram showing the relative positional relationship between the head-mounted device and the display device of the doctor's console in one embodiment of the present invention;

[0044] Figure 16 This is a flowchart illustrating the control method of a surgical robot system according to an embodiment of the present invention.

[0045] The accompanying figure is labeled as follows:

[0046] Head-mounted device - 110; Target object - 120; Magnetometer - 111; Receiver coil - 1111; Magnetic field generator - 200; Data transceiver - 112; Stand - 113; Magnetic field lines - 210; Transmitting coil - 220; Line of sight orientation - 300; Doctor's console - 10; Operating trolley - 20; Side trolley - 30; Robotic arm - 31; Main control arm - 11; Display device - 12; Processor - 131; Communication interface - 132; Memory - 133; Communication bus - 134. Detailed Implementation

[0047] The following is in conjunction with the appendix Figures 1 to 16The method for detecting the relative relationship between the line of sight and the target object, the surgical robot system, the control method, and the storage medium proposed in this invention will be further described in detail with specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, only for the purpose of conveniently and clearly illustrating the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0048] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] The main objective of this invention is to provide a method, a surgical robot system, a control method, and a storage medium for detecting the relative relationship between the line of sight and the target object, which can effectively determine whether the doctor is looking at the display device (target object) to ensure the safety of the surgery.

[0050] To achieve the above objectives, this invention provides a method for detecting the relative relationship between a line of sight and a target object. Please refer to... Figure 1 The diagram illustrates a flowchart of a method for detecting the relative relationship between a line of sight and a target object according to an embodiment of the present invention. Figure 1 As shown, the method for detecting the relative relationship between the line of sight and the target object includes the following steps:

[0051] Step S11: Obtain the relative positional relationship between the subject's face and the target object;

[0052] Step S12: Based on the relative positional relationship between the subject's face and the target object, obtain the subject's gaze orientation range;

[0053] Step S13: Determine whether the range of the subject's line of sight at least partially covers the target object;

[0054] If so, then it is determined that the subject's line of sight is facing the target object;

[0055] If not, issue a warning message and / or issue a status change command.

[0056] Therefore, compared to existing methods for detecting the relative relationship between a line of sight and a target object, this invention has lower requirements for the positional stability of the subject during the line-of-sight detection process. The line-of-sight detection will not fail due to the subject deviating from the predetermined testing range for any reason, greatly improving the operability of line-of-sight detection and reducing the false judgment rate. Furthermore, when this invention determines that the subject's line of sight is not facing the target object, it automatically issues a warning message and / or a status change command, thereby effectively preventing misoperation and improving safety during operation.

[0057] Please refer to Figure 2 It schematically illustrates a diagram of a test subject's line of sight facing a target object according to an embodiment of the present invention, such as... Figure 2 As shown, the subject's face may be covered by a head-mounted device 110, and the acquisition of the relative positional relationship between the subject's face and the target object includes:

[0058] Obtain the relative positional relationship between the head-mounted device and the target object;

[0059] The relative positional relationship between the subject's face and the target object is obtained based on the relative positional relationship between the head-mounted device and the target object, and the relative positional relationship between the subject's face and the head-mounted device.

[0060] Since the head-mounted device 110 is worn on the head of the subject, that is, there is a predetermined relative positional relationship between the subject's face and the head-mounted device 110, the relative positional relationship between the subject's face and the target object 120 can be obtained based on the relative positional relationship between the head-mounted device 110 and the target object 120.

[0061] Furthermore, obtaining the relative positional relationship between the head-mounted device and the target object includes:

[0062] Obtain the pose information of the head-mounted device in the target object coordinate system.

[0063] In some embodiments, the target object coordinate system is established with the center of the target object as the origin. Of course, in other embodiments, the target object coordinate system can also be established with other positions on the target object as the origin, depending on actual needs. This invention does not limit this. Therefore, this embodiment can more conveniently and quickly obtain the relative positional relationship between the head-mounted device 110 and the target object 120 by obtaining the three-dimensional coordinates of the head-mounted device 110 in the coordinate system of the target object 120. It should be noted that, as those skilled in the art will understand, in other embodiments, the three-dimensional coordinates of the head-mounted device 110 and the target object 120 can also be obtained separately in the same coordinate system, and then the relative positional relationship between the head-mounted device 110 and the target object 120 can be obtained based on kinematic equations. Regarding how to obtain the three-dimensional coordinates of the head-mounted device 110 and the target object 120 separately in the same coordinate system, existing position measurement techniques, such as binocular vision measurement, can be referenced, and this invention will not elaborate further.

[0064] For further details, please refer to... Figures 2 to 4 ,in Figure 3 A schematic diagram illustrating the relative positional relationship between the target object and the head-mounted device 110 in the X / Y directions according to an embodiment of the present invention is provided. Figure 4 A schematic diagram illustrating the relative positional relationship between the target object 120 and the head-mounted device 110 in the X / Z directions according to an embodiment of the present invention is provided. Figures 2 to 4 As shown, the head-mounted device 110 is equipped with multiple magnetic sensors 111, and the target object 120 is equipped with a magnetic field generator 200. The magnetic field generator 200 is used to generate a magnetic field within a predetermined range, and the magnetic sensors 111 are used to induce a corresponding current in the magnetic field. The coordinates of the magnetic sensors 111 in the magnetic field coordinate system can be obtained based on the induced current. Since there is a predetermined positional relationship between the magnetic sensors 111 and the head-mounted device 110, and between the magnetic field generator 200 and the target object 120, the pose information of the head-mounted device 110 in the target object coordinate system can be obtained by acquiring the three-dimensional coordinates of the multiple magnetic sensors 111 in the magnetic field coordinate system, and by using the mapping relationship between the magnetic field coordinate system and the target object coordinate system. Thus, the relative positional relationship between the head-mounted device 110 and the target object 120 can be obtained.

[0065] In some embodiments, the magnetic field generator 200 is mounted on the central axis of the target object 120. This arrangement facilitates obtaining the mapping relationship between the magnetic field coordinate system and the target object coordinate system. It should be noted that, as those skilled in the art will understand, in other embodiments, the magnetic field generator 200 may also be mounted on other components having a predetermined positional relationship with the target object 120. To improve measurement accuracy, the magnetic field coordinate system in this embodiment needs to be calibrated beforehand to obtain an accurate magnetic field.

[0066] In some embodiments, such as Figure 3 and 4 As shown, the head-mounted device 110 is a 3D glasses device, which includes a frame. Multiple magnetic sensors 111 are mounted on the support 113 of the frame. Therefore, the pose information of the frame can be obtained based on the measured three-dimensional coordinates of the multiple magnetic sensors 111. Based on the pose information of the frame relative to the target object 120, the relative positional relationship between the subject's face and the target object 120 can be obtained.

[0067] Furthermore, such as Figure 3 As shown, at least two of the plurality of magnetic sensors 111 are located on the brackets 113 on different sides of the eyeglasses frame. Because at least two of the plurality of magnetic sensors 111 are located on the brackets 113 on different sides of the eyeglasses frame, the pose information of the positions on the eyeglasses frame corresponding to the left and right eyes of the subject can be measured simultaneously, thereby making it easier to obtain the subject's gaze orientation range 300 (e.g., ...). Figure 2 As shown), the test subject's gaze direction interval 300 is determined to be facing the target object 120 based on whether the test subject's gaze direction interval 300 at least covers the target object 120, that is, based on whether the test subject's gaze direction interval 300 at least partially overlaps with the target object 120.

[0068] In some embodiments, such as Figure 3As shown, the head-mounted device 110 is equipped with four magnetic sensors 111A, 111B, 111C, and 111D. Magnetic sensors 111A and 111B are located on the left support 113A of the eyeglasses frame of the head-mounted device 110, while magnetic sensors 111C and 111D are located on the right support 113B of the eyeglasses frame. Therefore, the pose information of the left side of the subject's face can be obtained using the three-dimensional coordinates of the two magnetic sensors 111A and 111B located on the left support 113A of the eyeglasses frame, and the pose information of the right side of the subject's face can be obtained using the three-dimensional coordinates of the magnetic sensors 111C and 111D located on the right support 113B of the eyeglasses frame. This facilitates the acquisition of the subject's gaze orientation range 300.

[0069] like Figure 3 As shown, the head-mounted device 110 is equipped with a data transceiver 112 to facilitate data transmission. For example, it sends the induced current information of each magnetic sensor to a controller located on or independently of the head-mounted device. The controller calculates the subject's gaze orientation range 300 based on the induced current information. More preferably, the data transceiver 112 is a Bluetooth module. It should be noted that, as those skilled in the art will understand, the data transceiver 112 can also be other components capable of data transmission besides a Bluetooth module, such as components capable of wired or wireless transmission. This invention does not impose any limitations on this.

[0070] The following is through Figures 5 to 9 The measurement principles of the magnetic sensor 111 and the magnetic field generator 200 in this embodiment will be explained, wherein... Figure 5 A schematic diagram illustrating the working principle of a magnetic field generator 200 provided in an embodiment of the present invention is shown. Figure 6 A schematic diagram of a magnetic field coordinate system provided in an embodiment of the present invention is shown. Figure 7 A schematic diagram illustrating the working principle of a magnetic sensor 111 provided in an embodiment of the present invention is shown. Figure 8 A schematic diagram illustrating the coordinate measurement principle provided by an embodiment of the present invention is shown. Figure 9 A schematic diagram illustrating the coordinate measurement results of a magnetic sensor 111 provided in an embodiment of the present invention is shown. For example... Figure 5 As shown, the magnetic field generator 200 includes a transmitting coil 220. When different intensities of current / alternating current are passed through the transmitting coil 220, it will generate magnetic fields of different intensities / uniformly distributed in space. If a current varying at a fixed frequency f0 is input to the magnetic field generator 200, the magnetic field generator 200 will generate a magnetic field varying according to the fixed frequency f0. Furthermore, as... Figure 5 and Figure 6 As shown, since the main component of the magnetic field generator 200 is the transmitting coil 220, the magnetic field generated by the magnetic field generator 200 is not a uniformly distributed magnetic field in space, but a gradually varying magnetic field with the center of the transmitting coil 220 as the center of symmetry. In this embodiment, the origin of the magnetic field coordinate system is defined as the center of the transmitting coil 220. Since each point in this magnetic field coordinate system has different magnetic field strengths in the three directions (X, Y, Z), the coordinates of each point in this magnetic field coordinate system can be obtained based on this principle, thereby obtaining the position information of that point.

[0071] like Figure 7 and Figure 8 As shown, the magnetic sensor 111 includes three mutually orthogonal receiving coils 1111. When the magnetic field generator 200 generates a magnetic field that varies at a fixed frequency f0, and the magnetic field lines 210 of this magnetic field pass through the three receiving coils 1111 of the magnetic sensor 111, since the magnetic flux through the three receiving coils 1111 is different, an induced current I is generated in each of the three coils. x I y I z According to the induced current I x I y I z The three-dimensional coordinates of the magnetic sensor 111 in the magnetic field coordinate system can then be calculated. For example... Figure 9 As shown, the three-dimensional coordinates of magnetic sensor 111A are (a,b,c), the three-dimensional coordinates of magnetic sensor 111B are (l,m,n), the coordinates of magnetic sensor 111C are (d,e,f), and the three-dimensional coordinates of magnetic sensor 111D are (g,h,i). Since the area enclosed by magnetic sensors 111A, 111B, 111C, and 111D represents the area where the subject's left and right eyes are located, the subject's gaze orientation interval 300 can be determined based on the area enclosed by magnetic sensors 111A, 111B, 111C, and 111D.

[0072] In some embodiments, the method for obtaining the three-dimensional coordinates of the plurality of magnetic sensors in the magnetic field coordinate system includes:

[0073] Based on the pre-obtained position calibration results, the three-dimensional coordinates of the multiple magnetic sensors 111 in the magnetic field coordinate system are obtained respectively.

[0074] The pre-acquired position calibration result is the correspondence between the induced current and the position coordinates, so that the three-dimensional coordinates of the magnetic sensor 111 in the magnetic field coordinate system can be obtained based on the induced current generated by the three receiving coils 1111 in the magnetic sensor 111. Specifically, the position calibration result is obtained by calibrating the position of the magnetic sensor 111 relative to the magnetic field generator 200.

[0075] The target object in the method for detecting the relative relationship between the line of sight and the target object described above is not particularly limited. For example, it can be a display device or other components besides a display device. It can be used in various scenarios for identifying whether the line of sight is within a predetermined range. According to one aspect of the present invention, its application in a surgical robot system will be described below as an example. It should be noted that although the present invention uses a display device as the target object and a surgical robot system as the application scenario for description, this should not be construed as limiting the present invention.

[0076] Please refer to Figure 10 and Figure 11 ,in, Figure 10 A schematic diagram of the overall structure of a surgical robot system provided in an embodiment of the present invention is shown. Figure 11 A schematic diagram of the structure of a doctor's control console according to an embodiment of the present invention is shown. Figure 10 and Figure 11As shown, the surgical robot system includes a control end and an execution end. The control end includes a doctor's console 10 equipped with a main control arm 11, and the execution end includes equipment such as a surgical trolley 20 and a side trolley 30, where the patient lies on the surgical trolley 20 for surgery. The side trolley 30 is equipped with a robotic arm 31 for mounting surgical instruments and endoscopes. The robotic arm 31, surgical instruments, endoscopes, and the main control arm 11 have a predetermined mapping relationship, thus forming a master-slave relationship. After the surgical instruments are connected to the robotic arm 31, the system performs actions on the surgical instruments in various directions based on the movement of the main control arm 11 to complete the surgery. The doctor's console 10 includes a multi-axis robotic arm (i.e., the main control arm 11), a display device 12 (target object), a head-mounted device 110, and a controller. The display device 12 is communicatively connected to the controller. The doctor (subject) remotely controls the robotic arm 31 to perform surgical operations by operating the multi-axis robotic arm. The display device 12 can display the intraoperative process obtained from the endoscope in the abdominal cavity when the robotic arm 31 is performing surgery. The side trolley 30 includes two or more robotic arms 31. The doctor controls two of the robotic arms 31 through the multi-axis robotic arm (i.e., the main control arm 11) of the doctor's console 10. The actions of these two robotic arms 31 in operating surgical instruments (such as clamping and removing lesions) can be captured by the camera of the endoscope and displayed on the display device 12 of the doctor's console 10.

[0077] Please refer to Figure 12 The diagram illustrates the block structure of the controller in this embodiment, as shown below. Figure 12 As shown, the controller includes a processor 131 and a memory 133. The memory 133 stores a computer program, which, when executed by the processor 131, performs the following steps:

[0078] Obtain the relative positional relationship between the subject's face and the display device;

[0079] Based on the relative positional relationship between the subject's face and the display device, the subject's gaze orientation range is obtained;

[0080] Determine whether the subject's line of sight at least partially covers the display device;

[0081] If so, then it is determined that the subject's line of sight is facing the display device;

[0082] If not, issue a warning message and / or lock the robotic arm.

[0083] Therefore, compared to methods that detect gaze by collecting eye movement data, this invention has lower requirements for the positional stability of the subject during the gaze detection process. The gaze detection will not fail due to the subject deviating from the predetermined testing range for any reason, greatly improving the operability and accuracy of gaze detection. It can effectively determine whether the doctor (subject) is looking at the display device 12 (target object), ensuring the safety of the surgery. Furthermore, if the determination result is that the doctor's gaze is not facing the display device 12, the controller issues an alarm message and / or locks the robotic arm 31, thereby effectively preventing misoperation and improving the safety of the surgical process.

[0084] Regarding how to obtain the relative positional relationship between the subject's face and the display device 12, please refer to the relevant description in the method for detecting the relative relationship between the line of sight and the target object described above. This embodiment will not repeat the details.

[0085] Please refer to Figures 13 to 15 ,in Figure 13 A schematic diagram of the coordinate system of a display device 12 provided in an embodiment of the present invention is shown; Figure 14 A schematic diagram illustrating the relative positional relationship between a magnetic field generator 200 and a display device 12 of a doctor's control console 10 according to an embodiment of the present invention is provided. Figure 15 A schematic diagram illustrating the relative positional relationship between a head-mounted device 110 and a display device 12 of a doctor's control console 10 according to an embodiment of the present invention is provided. Figures 13 to 15As shown, during gaze detection, the doctor (i.e., the subject) wears the head-mounted device 110, which is communicatively connected to the controller and maintains a normal operating posture. The medical control system is enabled, and the magnetic field generator 200 (which can be mounted on the display device 12 or on other components) generates a magnetic field that changes at a fixed frequency under the action of alternating current. The magnetic field lines 210 of the magnetic field pass through the receiving coil 1111 of the magnetic sensor 111 on the head-mounted device 110, generating an induced current. The data transceiver 112 on the head-mounted device 110 transmits the induced current information to the controller. The controller calculates the coordinate values ​​corresponding to each induced current based on pre-stored position calibration results. This allows us to obtain the three-dimensional coordinates of each magnetic sensor 111 in the magnetic field coordinate system. Since the magnetic field generator 200 and the display device 12 have a predetermined positional relationship, the relative positional relationship between the subject's face and the display device 12 can be obtained based on the three-dimensional coordinates of each magnetic sensor 111 in the magnetic field coordinate system and the mapping relationship between the magnetic field coordinate system and the display device coordinate system. Based on the relative positional relationship between the subject's face and the display device 12, the subject's gaze orientation range can be obtained. By determining whether the subject's gaze orientation range at least partially covers the display device 12, that is, whether the subject's gaze orientation range at least partially coincides with the display device 12, we can determine whether the doctor's gaze is facing the display device 12. If the doctor's gaze is directed towards the display device 12, a master-slave relationship is established between the main control arm 11 and the robotic arm 31, entering a master-slave connection state. The doctor can then operate the main control arm 11 normally and control the robotic arm 31 to perform surgery. If the doctor's gaze is not directed towards the display device 12, the system automatically issues an alarm and / or activates a protection mechanism. The main control arm 11 cannot operate the robotic arm 31 to perform surgery, entering a locked state. The doctor needs to lightly touch the screen unlock button, pinch the end joint of the main control arm 11, or perform other unlocking actions to re-detect the gaze. Once the doctor's gaze is directed towards the display device 12 as described above, the robotic arm 31 is unlocked, and the doctor can perform surgery normally. Therefore, by using the surgical robot system provided by this invention, the doctor's gaze can be successfully detected, ensuring that the master-slave control relationship between the main control arm 11 and the robotic arm 31 can only be controlled when the doctor's gaze is directed towards the display device 12; otherwise, the robotic arm 31 will be automatically locked. This effectively avoids misoperation and improves safety during surgery.

[0086] like Figure 12As shown, the controller also includes a communication interface 132 and a communication bus 134, wherein the processor 131, the communication interface 132, and the memory 133 communicate with each other via the communication bus 134. The communication bus 134 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 134 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 132 is used for communication between the controller and other devices.

[0087] The processor 131 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor 131, etc. The processor 131 is the control center of the controller, connecting various parts of the entire controller via various interfaces and lines.

[0088] The memory 133 can be used to store the computer program, and the processor 131 implements various functions of the controller by running or executing the computer program stored in the memory 133 and calling the data stored in the memory 133.

[0089] The memory 133 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0090] Corresponding to the surgical robot system described above, this invention also provides a control method for a surgical robot system, please refer to [reference needed]. Figure 16 The diagram illustrates a flowchart of a control method for a surgical robot system according to an embodiment of the present invention, as shown below. Figure 16 As shown, the control method of the surgical robot system includes the following steps:

[0091] Step S21: Obtain the relative positional relationship between the subject's face and the display device;

[0092] Step S22: Based on the relative positional relationship between the subject's face and the display device, obtain the subject's gaze orientation range;

[0093] Step S23: Determine whether the range of the subject's line of sight at least partially covers the display device;

[0094] If so, then it is determined that the subject's line of sight is facing the display device;

[0095] If not, an alarm message will be issued and / or the robotic arm will be locked.

[0096] Specifically, if the judgment result indicates that the doctor's (i.e., the subject's) gaze is facing the display device 12, a master-slave relationship is established between the main control arm 11 and the robotic arm 31, entering a master-slave connection state. The doctor can then operate the main control arm 11 normally and control the robotic arm 31 to perform surgery. If the judgment result indicates that the doctor's gaze is not facing the display device 12, the system automatically issues an alarm message and / or activates a protection mechanism. The main control arm 11 cannot operate the robotic arm 31 to perform surgery, entering a locked state. The doctor needs to lightly touch the screen unlock button, pinch the end joint of the main control arm 11, or perform other unlocking actions to re-detect the gaze. Once the judgment result, as described above, indicates that the doctor's gaze is facing the display device 12, the robotic arm 31 is unlocked, and the doctor can perform surgery normally. Therefore, this invention successfully detects the doctor's gaze, ensuring that the master-slave control relationship between the main control arm 11 and the robotic arm 31 can only be controlled when the doctor's gaze is facing the display device 12; otherwise, the robotic arm 31 is automatically locked, effectively preventing misoperation and improving safety during surgery.

[0097] Regarding how to obtain the relative positional relationship between the subject's face and the display device 12, please refer to the relevant description in the method for detecting the relative relationship between the line of sight and the target object above. This embodiment will not repeat the details.

[0098] Based on the above-mentioned inventive concepts, this invention also provides a readable storage medium storing a computer program. When executed by a processor, the computer program can implement the method for detecting the relative relationship between the gaze and the target object, or the control method of a surgical robot system, as described above. Thus, the storage medium provided by this invention first acquires the relative positional relationship between the subject's face and the target object (e.g., a display device); then, based on this relative positional relationship, it acquires the subject's gaze orientation range; finally, it determines whether the subject's gaze orientation range at least partially covers the target object (e.g., the display device), i.e., whether the subject's gaze orientation range at least partially overlaps with the target object. If so, it determines that the subject's gaze is facing the target object (e.g., the display device); otherwise, it issues a warning message and / or a state change command. Therefore, compared to methods that detect gaze by collecting eye movement data, this invention has lower requirements for the positional stability of the subject during the gaze detection process. It will not fail to complete the gaze detection because the subject deviates from the predetermined test range for some reason, greatly improving the operability and accuracy of gaze detection. Furthermore, when the present invention determines that the subject's gaze is not directed towards the target object, it automatically issues a warning message and / or a status change command, thereby effectively preventing misoperation and improving safety during operation. For example, in a surgical robot system, the storage medium provided by the present invention automatically issues a warning message and / or activates a protection mechanism when it determines that the doctor's (subject's) gaze is not directed towards the display device (target object), causing the robotic arm used to perform the surgery to enter a locked state. This effectively prevents the doctor from operating the robotic arm when their gaze is not directed towards the display device, further improving safety during the surgical process and effectively preventing misoperation.

[0099] The readable storage medium of this invention can be any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer hard disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0100] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0101] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] In summary, compared with existing technologies, the method, surgical robot system, control method, and storage medium for detecting the relative relationship between the gaze and the target object provided by this invention have the following advantages: This invention first acquires the relative positional relationship between the subject's face and the target object (e.g., a display device); then, based on this relative positional relationship, it acquires the subject's gaze orientation range; finally, it determines whether the subject's gaze orientation range at least partially covers the target object (e.g., the display device), i.e., whether the subject's gaze orientation range at least partially overlaps with the target object. If so, it is determined that the subject's gaze is facing the target object (e.g., the display device). Therefore, compared with methods that achieve gaze detection by collecting eye movement data, this invention has lower requirements for the positional stability of the subject during the gaze detection process. It will not fail to complete the gaze detection because the subject deviates from the predetermined test range for some reason, greatly improving the operability and accuracy of gaze detection. Furthermore, when this invention determines that the subject's gaze is not facing the target object, it automatically issues a warning message and / or a status change command, thereby effectively preventing misoperation and improving safety during operation. For example, in the case of a surgical robot system, when the present invention determines that the doctor's (subject's) line of sight is not facing the display device (target object), it will automatically issue a warning message and / or activate a protection mechanism to lock the robotic arm used to perform the surgery. This can effectively prevent the doctor from operating the robotic arm when his line of sight is not facing the display device, further improving the safety of the surgical process and effectively preventing misoperation.

[0103] It should be noted that the apparatuses and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0105] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the scope of the claims. Obviously, those skilled in the art can make various modifications and variations to the invention without departing from the spirit and scope of the invention. Therefore, if these modifications and variations of the invention fall within the scope of the claims and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the following steps: Obtain the relative positional relationship between the subject's face and the target object; Based on the relative positional relationship between the subject's face and the target object, the subject's gaze orientation range is obtained; Determine whether the range of the subject's line of sight at least partially covers the target object; If not, issue a warning message and / or issue a status change command; The subject wears a head-mounted device equipped with multiple magnetic sensors, and the target object is equipped with a magnetic field generator. The process of acquiring the relative positional relationship between the subject's face and the target object includes: The three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system generated by the magnetic field generator are obtained respectively. Based on the three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system and the mapping relationship between the magnetic field coordinate system and the target object coordinate system, the relative positional relationship between the head-mounted device and the target object is obtained; The relative positional relationship between the subject's face and the target object is obtained based on the relative positional relationship between the head-mounted device and the target object, and the relative positional relationship between the subject's face and the head-mounted device.

2. The readable storage medium according to claim 1, characterized in that, When the computer program is executed by the processor, it also performs the following steps: If the range of the subject's gaze at least partially covers the target object, then it is determined that the subject's gaze is facing the target object.

3. The readable storage medium according to claim 1, characterized in that, The magnetic field generator is installed on the central axis of the target object.

4. The readable storage medium according to claim 1, characterized in that, The step of obtaining the three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system includes: Based on the pre-obtained position calibration results, the three-dimensional coordinates of the multiple magnetic sensors in the magnetic field coordinate system are obtained respectively.

5. The readable storage medium according to claim 1, characterized in that, The target object is a display device, and the step of determining whether the subject's line of sight at least partially covers the target object includes: Determine whether the range of the subject's gaze direction at least partially overlaps with the target object.

6. The readable storage medium according to claim 2, characterized in that, The target object is a display device, and when the computer program is executed by the processor, it also performs the following steps: If it is determined that the subject's gaze is not directed toward the display device, a warning message is issued and / or the robotic arm of the surgical robot system is locked.

7. A surgical robot system, characterized in that, The device includes a display device, a controller, and a robotic arm. The display device is communicatively connected to the controller. The controller includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the steps that can be implemented by a readable storage medium as described in any one of claims 1 to 6.

8. The surgical robot system according to claim 7, characterized in that, The surgical robot system also includes the head-mounted device, which is communicatively connected to the controller. The head-mounted device includes an eyeglass frame, and the frame of the eyeglass frame is provided with multiple magnetic sensors. The display device is provided with a magnetic field generator, which is used to generate a magnetic field within a predetermined range. The magnetic sensors are used to induce a corresponding current in the magnetic field. The controller is used to calculate the relative positional relationship between the head-mounted device and the display device based on the induced current.

9. The surgical robot system according to claim 8, characterized in that, At least two of the plurality of magnetic sensors are located on the support on different sides of the eyeglass frame.

10. The surgical robot system according to claim 7, characterized in that, The surgical robot system includes a doctor's console and a side trolley. The doctor's console includes a display device, a controller, and a main control arm. The side trolley includes the robotic arm. The controller is used to determine whether the subject's gaze is directed towards the display device, and then establish a master-slave relationship between the main control arm and the robotic arm.

11. The surgical robot system according to claim 8, characterized in that, The magnetic field generator includes a transmitting coil, and the magnetic sensor includes three mutually orthogonal receiving coils.

Images