Robotic arm control method, device, system and storage medium

Abstract

The present application relates to a robotic arm control method, device, system, storage medium, and computer program product. The method comprises: collecting real-time eye video of a target object gazing at a display device; determining the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video; determining a robotic arm adjustment mode based on the real-time gaze point position; if the robotic arm adjustment mode is an endoscope adjustment mode, controlling the movement of the endoscope robotic arm based on the real-time gaze point position until the endoscope angle reaches a preset angle, and updating the robotic arm adjustment mode to a tool adjustment mode; if the robotic arm adjustment mode is a tool adjustment mode, obtaining a current captured image of the endoscope; determining a target area in the current captured image based on the real-time gaze point position; and controlling the movement of the tool robotic arm based on the target area. This method can improve the efficiency of robotic arm control.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot arm control method, device, system, storage medium and computer program product. Background Art

[0002] Existing robotic arm control methods often involve manually specifying a target area and then controlling the robotic arm to move into the target area. However, this robotic arm control method suffers from low robotic arm control efficiency in scenarios where the target area needs to be updated in real time based on actual conditions. Summary of the Invention

[0003] Based on this, it is necessary to provide a robotic arm control method, device, system, computer-readable storage medium and computer program product that can improve the efficiency of robotic arm control in order to address the problem of low robotic arm control efficiency in existing robotic arm control methods.

[0004] In a first aspect, the present application provides a method for controlling a robotic arm. The method comprises:

[0005] Collecting real-time eye video of the target object looking at a display device, and the display device is used to display the real-time image collected by the endoscope of the area to be tested;

[0006] According to the real-time eye video, the real-time gaze point position of the target object in the display device is determined by eye tracking;

[0007] Determine the robot arm adjustment mode based on the real-time gaze point position;

[0008] When the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches the preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode;

[0009] When the manipulator adjustment mode is the tool adjustment mode, the current captured image of the endoscope is acquired; the target area is determined in the current captured image according to the real-time gaze point position; and the movement of the tool manipulator is controlled according to the target area.

[0010] In one embodiment, controlling the movement of an endoscope robotic arm according to a real-time gaze point position includes:

[0011] determining whether the real-time gaze point position is within an endoscope adjustment area of a display device;

[0012] When the real-time gaze point position is within the endoscope adjustment area, the rotation angle is determined according to the position of the endoscope adjustment area in the display device; and the movement of the endoscope robotic arm is controlled based on the rotation angle.

[0013] In one embodiment, determining a target area in a currently captured image based on a real-time gaze point position includes:

[0014] Obtain the real-time fixation point location’s attention duration and number of fixations;

[0015] Determine the real-time area centered on the real-time gaze point position based on the attention duration and the number of attention times;

[0016] Within the preset time period, when the distance between the real-time gaze point positions at any two moments is within the preset area, the real-time area corresponding to the real-time gaze point positions in the current captured image is determined as the target area.

[0017] In one embodiment, controlling the movement of a tool robot arm according to a target area includes:

[0018] According to the position of the target area in the current acquired image, the position of the end of the tool manipulator in the robot coordinate system is obtained through the conversion relationship between the display coordinate system and the robot coordinate system;

[0019] According to the end position of the instrument, the twisting amount of each joint of the tool manipulator is determined through inverse kinematics;

[0020] Control the tool robot arm to move according to the twisting amount of each joint.

[0021] In one embodiment, determining the real-time gaze point position of a target object on a display device through eye tracking based on real-time eye video includes:

[0022] According to the real-time eye video, the pupil position and Purkinje spot position are obtained through pupil recognition;

[0023] According to the pupil position and Purkinje spot position, the real-time gaze point position of the target object in the display device is determined by calibrating the mapping function.

[0024] In one embodiment, controlling the movement of a tool robot arm according to a target area includes:

[0025] Receive robotic arm control instructions;

[0026] When the robot arm control instruction is a zoom instruction, the target area is zoomed in and the zoomed target area is displayed on the display device; and the tool robot arm is controlled to move according to the zoomed target area;

[0027] When the robot arm control instruction is not an amplification instruction, the tool robot arm is controlled to move according to the target area.

[0028] In a second aspect, the present application further provides a robotic arm control device. The device comprises:

[0029] An acquisition module is used to acquire real-time eye video of the target object looking at a display device, and the display device is used to display the real-time acquired image of the endoscope on the area to be measured;

[0030] An eye tracking module is used to determine the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video;

[0031] A mode determination module is used to determine the adjustment mode of the robotic arm based on the real-time gaze point position;

[0032] a first control module, configured to control the movement of the endoscope manipulator arm according to the real-time gaze point position, when the manipulator arm adjustment mode is the endoscope adjustment mode, until the endoscope angle reaches a preset angle, and update the manipulator arm adjustment mode to the tool adjustment mode;

[0033] The second control module is used to obtain the current captured image of the endoscope when the robot arm adjustment mode is the tool adjustment mode; determine the target area in the current captured image according to the real-time gaze point position; and control the movement of the tool robot arm according to the target area.

[0034] In a third aspect, the present application further provides a robotic arm control system, the system comprising: a robotic arm, including an endoscope robotic arm and a tool robotic arm; an image acquisition device, for acquiring real-time eye video of a target object looking at a display device; an endoscope, the endoscope being mounted at the end of the endoscope robotic arm and for acquiring real-time images of the area to be measured; a display device, for displaying the real-time images of the area to be measured acquired by the endoscope; and a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0035] Collecting real-time eye video of the target object looking at a display device, and the display device is used to display the real-time image collected by the endoscope of the area to be tested;

[0036] According to the real-time eye video, the real-time gaze point position of the target object in the display device is determined by eye tracking;

[0037] Determine the robot arm adjustment mode based on the real-time gaze point position;

[0038] When the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches the preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode;

[0039] When the manipulator adjustment mode is the tool adjustment mode, the current captured image of the endoscope is acquired; the target area is determined in the current captured image according to the real-time gaze point position; and the movement of the tool manipulator is controlled according to the target area.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0041] Collecting real-time eye video of the target object looking at a display device, and the display device is used to display the real-time image collected by the endoscope of the area to be tested;

[0042] According to the real-time eye video, the real-time gaze point position of the target object in the display device is determined by eye tracking;

[0043] Determine the robot arm adjustment mode based on the real-time gaze point position;

[0044] When the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches the preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode;

[0045] When the manipulator adjustment mode is the tool adjustment mode, the current captured image of the endoscope is acquired; the target area is determined in the current captured image according to the real-time gaze point position; and the movement of the tool manipulator is controlled according to the target area.

[0046] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0047] Collecting real-time eye video of the target object looking at a display device, and the display device is used to display the real-time image collected by the endoscope of the area to be tested;

[0048] According to the real-time eye video, the real-time gaze point position of the target object in the display device is determined by eye tracking;

[0049] Determine the robot arm adjustment mode based on the real-time gaze point position;

[0050] When the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches the preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode;

[0051] When the manipulator adjustment mode is the tool adjustment mode, the current captured image of the endoscope is acquired; the target area is determined in the current captured image according to the real-time gaze point position; and the movement of the tool manipulator is controlled according to the target area.

[0052] The above-mentioned robotic arm control method, device, system, storage medium and computer program product collect real-time eye video of the target object looking at a display device, and the display device is used to display the real-time captured image of the endoscope on the area to be measured. Since the real-time gaze point position in the display device is obtained by eye tracking the real-time eye video of the target object, robotic arm control based on the real-time gaze point position is beneficial to improving the efficiency of robotic arm control; according to the real-time gaze point position, the robotic arm adjustment mode is determined. In the endoscope adjustment mode, the target object can update and adjust the angle of the endoscope in real time according to the actual situation observed by the endoscope, and automatically enter the tool adjustment mode when reaching the appropriate preset angle, which is beneficial to improving the control efficiency of the robotic arm; in the tool adjustment mode, the target area can be determined in the current captured image of the endoscope according to the real-time gaze point position, and then the movement of the tool robotic arm can be controlled. The determination of the robotic arm adjustment mode, the control of the endoscope robotic arm and the control of the tool robotic arm can be completed by the same target object by moving the eye line of sight, thereby improving the control efficiency of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A diagram showing an application environment of a robotic arm control method according to an embodiment;

[0054] Figure 2 1 is a flow chart of a method for controlling a robotic arm according to an embodiment;

[0055] Figure 3 is a schematic diagram of an endoscope adjustment area in one embodiment;

[0056] Figure 4 Schematic diagram of a sub-process of S205 in one embodiment;

[0057] Figure 5 Schematic diagram of a sub-process of S205 in another embodiment;

[0058] Figure 6 Schematic diagram of a sub-process of S202 in one embodiment;

[0059] Figure 7 Schematic diagram of a sub-process of S205 in another embodiment;

[0060] Figure 8 is a schematic diagram of receiving a robotic arm control instruction in one embodiment;

[0061] Figure 9 is a schematic diagram of magnifying a target area in one embodiment;

[0062] Figure 10 A schematic diagram of a method for performing calibration based on calibration points in one embodiment;

[0063] Figure 11 Schematic diagram of the composition of a robotic arm control system in one embodiment;

[0064] Figure 12 is a schematic diagram of a target object gazing at a display device in one embodiment;

[0065] Figure 13 1 is a schematic diagram of the overall flow of a method for controlling a robotic arm in one embodiment;

[0066] Figure 14 is a schematic diagram of a calibration process in one embodiment;

[0067] Figure 15 A schematic diagram of determining an adjustment mode of a robotic arm in one embodiment;

[0068] Figure 16 is a schematic diagram of exiting an endoscope adjustment mode in one embodiment;

[0069] Figure 17 A schematic diagram of determining a real-time area corresponding to a real-time gaze point position in one embodiment;

[0070] Figure 18 A schematic diagram of determining a target area in one embodiment;

[0071] Figure 19 1 is a schematic diagram of a process for controlling the movement of a tool robot arm according to a target area in one embodiment;

[0072] Figure 20 1. A schematic diagram of a process flow of ending control of a robotic arm in one embodiment;

[0073] Figure 21 is a schematic diagram of a robotic arm returning to an initial position in one embodiment;

[0074] Figure 22 is a structural block diagram of a robotic arm control device in one embodiment;

[0075] Figure 23 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0077] The robot arm control method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The manipulator control method is applied to a manipulator control system, and the system includes a manipulator 101, an image acquisition device 102, an endoscope 103, a display device 104, and a computer device 105. The manipulator 101 includes an endoscope manipulator 106 and a tool manipulator 107; the image acquisition device 102 is used to acquire real-time eye video of the target object looking at the display device 104; the endoscope 103 is installed at the end of the endoscope manipulator 106, and is used to acquire real-time images of the area to be measured; the display device 104 is used to display the real-time images of the area to be measured by the endoscope 103; the computer device 105 can be a terminal or a server. The manipulator control method provided in the embodiment of the present application can be executed by the terminal or the server alone, or by the terminal and the server in collaboration, and the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server, or placed on the cloud or other network servers. Taking the example of a terminal executing the process independently, the following steps are performed: A real-time eye video of the target subject gazing at the display device 104 is captured; the display device 104 is used to display a real-time image of the target area captured by the endoscope 103; based on the real-time eye video, the real-time gaze point position of the target subject on the display device 104 is determined through eye tracking; based on the real-time gaze point position, a robotic arm adjustment mode is determined; if the robotic arm adjustment mode is the endoscope adjustment mode, the endoscope robotic arm 106 is controlled to rotate based on the real-time gaze point position until the endoscope angle reaches a preset angle, and the robotic arm adjustment mode is updated to the tool adjustment mode; if the robotic arm adjustment mode is the tool adjustment mode, the current captured image of the endoscope 103 is acquired; based on the real-time gaze point position, a target area is determined in the current captured image; and based on the target area, the tool robotic arm 107 is controlled to move. The terminal may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0078] In one embodiment, Figure 2 As shown, a method for controlling a robotic arm is provided. Figure 1Taking the computer device 105 in the example as an example, the method includes the following steps:

[0079] S201 , collecting real-time eye video of a target object looking at a display device, where the display device is used to display a real-time image collected by an endoscope of a region to be measured.

[0080] The display device is a device with image or tactile output capabilities. Optionally, the display device can be a standalone device within the robotic arm control system or integrated into a computer. The endoscope captures images of the area to be tested and uses them as real-time captured images. The display device is used to display the real-time images of the area to be tested captured by the endoscope.

[0081] The computer device uses an image capture device to capture real-time eye video of a target subject gazing at a display device. For example, the display device displays a real-time image of the target area captured by an endoscope, and the target subject gazes at the real-time image displayed on the display device. The image capture device is positioned in front of the target subject, facing the target subject's eyes, to capture real-time eye video of the target subject gazing at the display device.

[0082] S202: Determine the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video.

[0083] Eye tracking measures the movement of a subject's eye gaze relative to their head. Eye tracking methods include observation, mechanical recording, electrooculography, electromagnetic induction, sclera-iris edge tracking, pupil tracking, and pupil-corneal reflection.

[0084] The real-time eye video includes at least one frame of real-time eye imagery, which forms an eye movement track. The computer performs eye tracking on this track to obtain the real-time gaze point location of the target subject on the display device. The real-time gaze point location represents the location on the display device where the target subject's gaze rests.

[0085] S203: Determine the adjustment mode of the robotic arm according to the real-time gaze point position.

[0086] Among them, the manipulator adjustment mode includes an endoscope adjustment mode and a tool adjustment mode. The computer device determines the manipulator adjustment mode based on the real-time gaze point position of the target object on the display device. In some embodiments, the display device can also display the content output by the computer device. The computer device outputs the position of each manipulator adjustment mode on the display device and displays it on the display device. The computer device determines the real-time gaze point position. If the real-time gaze point position is located at the position of the endoscope adjustment mode, the manipulator adjustment mode is determined to be the endoscope adjustment mode. If the real-time gaze point position is located at the position of the tool adjustment mode, the manipulator adjustment mode is determined to be the tool adjustment mode.

[0087] S204, when the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches the preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode.

[0088] The end of the endoscope arm is equipped with an endoscope, and the computer device controls the movement of the endoscope arm to adjust the rotation angle of the endoscope. The endoscope angle refers to the field of view of the endoscope.

[0089] When the manipulator adjustment mode is the endoscope adjustment mode, the computer device determines the endoscope rotation angle corresponding to the real-time gaze point position based on the real-time gaze point position and a pre-established relationship between the real-time gaze point position and the endoscope rotation angle, and controls the movement of the endoscope manipulator according to the endoscope rotation angle until the endoscope angle reaches a preset angle. When the endoscope angle reaches the preset angle, the manipulator adjustment mode is updated to the tool adjustment mode.

[0090] S205, when the manipulator adjustment mode is the tool adjustment mode, the current captured image of the endoscope is acquired; the target area is determined in the current captured image according to the real-time gaze point position; and the movement of the tool manipulator is controlled according to the target area.

[0091] The end of the tool manipulator arm is equipped with an execution tool, and the computer device controls the movement of the execution tool at the end of the tool manipulator arm by controlling the movement of the tool manipulator arm.

[0092] The current captured image refers to the endoscope captured image obtained when the computer device determines that the robot arm adjustment mode is the tool adjustment mode. The endoscope maintains the viewing angle corresponding to the current captured image, and the display device displays the current captured image.

[0093] The target area is an area in the current captured image. Optionally, the real-time gaze point position is within the target area, or the real-time gaze point position is at a preset distance from the target area. The computer device determines the target area in the current captured image based on the real-time gaze point position. The tool manipulator arm is controlled to move according to the target area. In some embodiments, the computer device controls the execution tool of the tool manipulator arm to move to any position in the target area. In other embodiments, the computer device controls the execution tool of the tool manipulator arm to move to the center position of the target area. In yet other embodiments, the computer device controls the execution tool of the tool manipulator arm to move to a position point that is a preset distance from the center position of the target area.

[0094] In the above-mentioned robotic arm control method, by collecting real-time eye video of the target object looking at the display device, the display device is used to display the real-time captured image of the endoscope on the area to be measured. Since the real-time gaze point position in the display device is obtained by eye tracking the real-time eye video of the target object, the robotic arm control based on the real-time gaze point position is beneficial to improving the efficiency of the robotic arm control; according to the real-time gaze point position, the robotic arm adjustment mode is determined. In the endoscope adjustment mode, the method of controlling the movement of the endoscope robotic arm according to the real-time gaze point position is used. The target object can update and adjust the angle of the endoscope in real time according to the actual situation observed by the endoscope, and automatically enter the tool adjustment mode when reaching the appropriate preset angle, which is beneficial to improving the control efficiency of the robotic arm; in the tool adjustment mode, the target area can be determined in the current captured image of the endoscope according to the real-time gaze point position, and then the movement of the tool robotic arm can be controlled. The determination of the robotic arm adjustment mode, the control of the endoscope robotic arm and the control of the tool robotic arm can be completed by the same target object by moving the eye line of sight, thereby improving the control efficiency of the robotic arm.

[0095] In one embodiment, the movement of the endoscope robotic arm is controlled according to the real-time gaze point position, including: determining whether the real-time gaze point position is located within the endoscope adjustment area in the display device; when the real-time gaze point position is located within the endoscope adjustment area, determining the rotation angle according to the position of the endoscope adjustment area in the display device; and controlling the movement of the endoscope robotic arm based on the rotation angle.

[0096] The endoscope adjustment area is an area within the display area of the display device. When the real-time gaze point position of the target object enters the endoscope adjustment area, the endoscope robotic arm can be controlled to move. The computer device determines whether the real-time gaze point position is within the endoscope adjustment area of the display device. When the real-time gaze point position is not within the endoscope adjustment area, the endoscope robotic arm is controlled not to move. When the real-time gaze point position is within the endoscope adjustment area, the rotation angle is determined based on the position of the endoscope adjustment area in the display device. The position of the endoscope adjustment area in the display device refers to the position of the endoscope adjustment area in the display area of the display device.

[0097] In some embodiments, the endoscope adjustment area includes at least one area. Figure 3 The figure shows a schematic diagram of the endoscope adjustment area within the display area of a display device, where the shaded area represents the endoscope adjustment area, which includes at least one area. When the real-time gaze point is located within any of the endoscope adjustment areas, a rotation angle is determined based on the position of the endoscope adjustment area corresponding to the real-time gaze point on the display device. Specifically, each endoscope adjustment area's position on the display device corresponds to a rotation angle, and the computer device determines the corresponding rotation angle based on the position of the endoscope adjustment area corresponding to the real-time gaze point on the display device.

[0098] The computer controls the movement of the endoscope's robotic arm based on the rotation angle. Specifically, the computer uses the rotation angle to calculate the twisting amount of each joint of the endoscope's robotic arm through inverse kinematics, and controls the movement of the endoscope's robotic arm according to the twisting amount of each joint.

[0099] In this embodiment, by determining whether the real-time gaze point position is within the endoscope adjustment area on the display device, and if the real-time gaze point position is within the endoscope adjustment area, the rotation angle is determined based on the position of the endoscope adjustment area on the display device, thereby controlling the movement of the endoscope manipulator. This method for controlling the endoscope manipulator improves the control efficiency of the endoscope manipulator by controlling the movement of the endoscope manipulator by determining whether the real-time gaze point position of the target object on the display device is within the endoscope adjustment area and thereby determining the rotation angle.

[0100] In one embodiment, Figure 4 As shown, according to the real-time gaze point position, the target area is determined in the current acquired image, including:

[0101] S402, obtaining the attention duration and the number of attention times of the real-time gaze point position.

[0102] The attention duration refers to the duration that the target object pays attention to the real-time gaze point location. The number of attentions refers to the number of times the target object pays attention to the real-time gaze point location. The computer device obtains the attention duration and the number of attentions of the real-time gaze point location.

[0103] S404: Determine a real-time area centered on the real-time gaze point position according to the attention duration and the number of attention times.

[0104] The computer device determines a real-time area centered on the real-time gaze point location based on the duration of attention and the number of times the attention is paid. The real-time area is centered on the real-time gaze point location. The area of the real-time area changes with the duration of attention and the number of times the attention is paid. In some embodiments, the longer the duration of attention, the larger the area of the real-time area, and the more times the attention is paid, the larger the area of the real-time area.

[0105] S406 , when the distance between the real-time gaze point positions at any two moments within the preset time period is within the preset area, the real-time area corresponding to the real-time gaze point positions in the current captured image is determined as the target area.

[0106] If the distance between the real-time gaze point positions at any two moments within a preset duration is within a preset region, this indicates that the real-time gaze position has only moved within the preset region and for a preset duration. The computer device determines the real-time region corresponding to the real-time gaze point position in the currently captured image as the target region. The target region is used to represent the target subject's region of interest relative to the target region.

[0107] In this embodiment, the real-time area centered on the real-time gaze point position is determined by the duration and number of times the real-time gaze point position is focused on. Within the preset time, when the distance between the real-time gaze point positions at any two moments is within the preset area, the real-time area corresponding to the real-time gaze point position in the current captured image is determined as the target area. This method for determining the target area is based on the real-time gaze point position of the target object, which improves the efficiency of obtaining the target object's area of interest to the test area, and is conducive to improving the control efficiency of the robotic arm.

[0108] In one embodiment, Figure 5 As shown, according to the target area, the tool robot arm movement is controlled, including:

[0109] S502 : According to the position of the target area in the current captured image, the position of the end of the tool manipulator in the robot coordinate system is obtained by converting the display coordinate system and the robot coordinate system.

[0110] The instrument end position refers to the position of the actuator at the end of the tool manipulator arm. The computer device establishes a display coordinate system based on the display area of the display device and establishes a robot coordinate system based on the tool manipulator arm. The position of the target area in the current acquired image is the position in the display coordinate system. The computer device multiplies the position of the target area in the current acquired image by the conversion relationship between the display coordinate system and the robot coordinate system to obtain the position in the robot coordinate system, and uses it as the instrument end position of the tool manipulator arm in the robot coordinate system. The instrument end position of the tool manipulator arm in the robot coordinate system is used to indicate the movement position of the actuator at the end of the tool manipulator arm.

[0111] S504 , determining the twisting amount of each joint of the tool manipulator according to the position of the end of the instrument through inverse kinematics; and controlling the tool manipulator to move according to the twisting amount of each joint.

[0112] Inverse kinematics is a method for converting pose data in the robot coordinate system into joint angles in joint space. A computer performs inverse kinematics calculations on the end-of-tool position to determine the torque of each joint in the tool arm. The computer then controls the movement of each joint in the tool arm according to the torque.

[0113] In this embodiment, the position of the tool manipulator's end effector in the robot coordinate system is determined by performing coordinate transformation on the target area within the currently captured image. Using inverse kinematics, the twisting motion of each joint of the tool manipulator is determined, and the tool manipulator is then controlled to move according to the twisting motion of each joint. This method, which controls the movement of the tool manipulator in real time based on the target area of the target object's line of sight, improves the control efficiency of the tool manipulator.

[0114] In one embodiment, Figure 6 As shown, according to the real-time eye video, the real-time gaze point position of the target object in the display device is determined through eye tracking, including:

[0115] S602: Obtain the pupil position and the Purkinje spot position through pupil recognition based on the real-time eye video.

[0116] Pupil recognition refers to a biometric technology that identifies the target subject based on the eye patterns on either side of the iris. Pupil position refers to the position of the target subject's pupil relative to the target subject's eye. Purkinje's spot refers to the glint produced on the target subject's cornea when light from a display device enters the target subject's eye. Purkinje's spot position refers to the position of the Purkinje's spot relative to the target subject's eye. A computer device uses pupil recognition to determine the pupil and Purkinje's spot positions for each frame of real-time eye video.

[0117] S604: Determine the real-time gaze point position of the target object on the display device by calibrating a mapping function according to the pupil position and the Purkinje spot position.

[0118] The calibration mapping function refers to the functional mapping relationship between the vector formed by the pupil position and the Purkinje spot position and the position point on the display device. The calibration mapping function is pre-calibrated. For each frame of real-time eye image, the computer device determines the vector formed by the pupil position and the Purkinje spot position in the current frame of real-time eye image, substitutes it into the calibration mapping function, and the resulting result is used as the real-time gaze point position of the target object on the display device.

[0119] In this embodiment, pupil recognition is performed on real-time eye video to obtain the pupil position and Purkinje spot position, and then the real-time gaze point position is determined through a pre-calibrated calibration mapping function. The real-time eye video of the target object collected can be converted into the real-time gaze point position on the display device, thereby realizing real-time tracking of the target object. The obtained real-time gaze point position is beneficial to improving the control efficiency of the robotic arm.

[0120] In one embodiment, Figure 7 As shown, according to the target area, the tool robot arm movement is controlled, including:

[0121] S702, receiving a robotic arm control instruction.

[0122] Among them, the robot arm control instruction refers to the control instruction input to the external controller or target object, which is used to control the movement of the robot arm. The computer device receives the robot arm control instruction input by the external controller or target object. The robot arm control instruction includes a zoom instruction and a maintain instruction. In some embodiments, the computer device outputs the position of the zoom instruction in the display device and the position of the maintain instruction in the display device, and displays the position of the zoom instruction and the position corresponding to the maintain instruction in the display area of the display device. When the real-time gaze point position of the target object is at the position of the zoom instruction, the robot arm control instruction received by the computer device is a zoom instruction. When the real-time gaze point position of the target object is not at the position of the zoom instruction, the robot arm control instruction received by the computer device is not a zoom instruction. Among them, when the real-time gaze point position of the target object is at the position of the maintain instruction, the robot arm control instruction received by the computer device is a maintain instruction. As Figure 8 Shown is a schematic diagram of receiving robot arm control instructions.

[0123] S704 , when the robot arm control instruction is a zoom instruction, the target area is zoomed in, and the zoomed target area is displayed on a display device; and the tool robot arm is controlled to move according to the zoomed target area.

[0124] Among them, when the robot arm control instruction is a zoom instruction, it indicates that the target object wants to enlarge the target area to achieve precise positioning of the robot arm. Optionally, the computer device enlarges the target area by a preset multiple, or the computer device enlarges the target area to a preset size. And the enlarged target area is displayed in the display area of the display device. Exemplarily, the enlarged target area can be displayed in the entire display area of the display device. The center point position of the target area in the display area of the display device can also be used as the center point position of the enlarged target area, and the enlarged target area can be displayed in the display area of the display device. Figure 9 Shown is a schematic diagram of a magnified target area.

[0125] The computer device controls the movement of the tool manipulator arm based on the magnified target area. For example, the computer device controls the execution tool at the end of the tool manipulator arm to move to any position within the magnified target area. Alternatively, the computer device controls the execution tool at the end of the tool manipulator arm to move to the center point of the magnified target area. Alternatively, the computer device controls the execution tool at the end of the tool manipulator arm to move to a position a preset distance from the center point of the magnified target area.

[0126] S706 , when the robot arm control instruction is not a zoom instruction, control the tool robot arm to move according to the target area.

[0127] Among them, when the robot arm control instruction is not a zoom instruction, it indicates that the target object does not need to enlarge the target area, the accuracy of the target area is sufficient, and the computer device controls the movement of the tool robot arm according to the target area.

[0128] In this embodiment, by receiving the robot arm control instruction, when the robot arm control instruction is a zoom instruction, the target area is enlarged and displayed, so that the movement of the tool robot arm is controlled according to the enlarged target area. This method of enlarging and displaying the target area to control the robot arm can improve the control accuracy of the robot arm, and for scenarios that require precise control, it can improve the control efficiency of the robot arm.

[0129] In some embodiments, the step of obtaining the calibration mapping function includes: collecting an eye video of the target object looking at a calibration point in a display device; obtaining the pupil position and Purkinje's spot position through pupil recognition based on the eye video; determining the calibration mapping function based on the pupil position, Purkinje's spot position and the calibration point position; the calibration mapping function is used to characterize the relationship between the vector composed of the pupil position and the Purkinje's spot position and the calibration point position.

[0130] The calibration points are easily distinguishable points displayed in the display area of the display device. The number of calibration points includes at least one. Figure 10Figure 1 shows a schematic diagram of the calibration method based on calibration points. The nine black dots in Figure (a) represent the calibration points, and the nine eye diagrams in Figure (b) represent the pupil positions and Purkinje spots of the target object when it is looking at the nine calibration points.

[0131] A computer device uses an image acquisition device to capture an eye video of a target subject gazing at a calibration point on a display device. Pupil recognition is performed on the eye image frames of the subject gazing at the calibration point in the eye video to obtain the pupil position and Purkinje spot position in the corresponding eye image frames. For each eye image frame corresponding to the calibration point, the computer device determines a vector consisting of the pupil position and Purkinje spot position corresponding to the current eye image frame. Based on this vector and the corresponding calibration point position, a calibration mapping function is determined. The calibration mapping function represents the relationship between the vector consisting of the pupil position and Purkinje spot position and the calibration point position.

[0132] To illustrate the control method and effect of the robot arm in this solution in detail, the following is a most detailed embodiment:

[0133] The robot arm control method is applied to the robot arm control system, and the robot arm control system includes: a robot arm, an image acquisition device, an endoscope, a display device, and a computer device. Among them, the robot arm includes an endoscope robot arm and a tool robot arm; the image acquisition device is used to acquire real-time eye video of the target object looking at the display device; the endoscope is installed at the end of the endoscope robot arm and is used to acquire real-time images of the area to be measured; the display device is used to display the real-time images of the area to be measured by the endoscope; the computer device includes a memory and a processor, the memory stores a computer program, and the robot arm control method is implemented when the processor executes the computer program. Figure 11 Shown is a schematic diagram of the composition of the robotic arm control system.

[0134] The computer device collects the real-time eye video of the target object looking at the display device, and the display device is used to display the real-time image collected by the endoscope of the area to be tested. Figure 12 The figure shows a target object looking at a display device. The image acquisition device is located directly in front of the target object, capturing real-time eye video of the target object looking at the display device, and the display device displays the real-time image captured by the endoscope of the area to be measured.

[0135] like Figure 13 The figure shows the overall flow of the robotic arm control method. Based on real-time eye video, the computer device uses eye tracking to determine the real-time gaze point location of the target object on the display device. Based on the real-time eye video, pupil recognition is performed to obtain the pupil position and Purkinje's spot position. Based on the pupil position and Purkinje's spot position, a calibration mapping function is used to determine the real-time gaze point location of the target object on the display device.

[0136] Among them, such as Figure 14 The figure shows a schematic diagram of the calibration process. The steps for obtaining the calibration mapping function include: using a computer to capture eye video of a target subject gazing at a calibration point on a display device; using the eye video to identify the pupil and the Purkinje spot; determining a calibration mapping function based on the pupil and Purkinje spot positions, and the calibration point positions; and using the calibration mapping function to represent the relationship between the vector formed by the pupil and Purkinje spot positions and the calibration point positions.

[0137] The computer device determines the adjustment mode of the robotic arm based on the real-time gaze point position. Figure 15 Shown is a schematic diagram for determining the adjustment mode of the robot arm.

[0138] When the manipulator adjustment mode is the endoscope adjustment mode, the computer device controls the movement of the endoscope manipulator according to the real-time gaze point position until the endoscope angle reaches the preset angle, and updates the manipulator adjustment mode to the tool adjustment mode. Controlling the movement of the endoscope manipulator according to the real-time gaze point position includes: determining whether the real-time gaze point position is located within the endoscope adjustment area of the display device; when the real-time gaze point position is located within the endoscope adjustment area, determining the rotation angle according to the position of the endoscope adjustment area in the display device; and controlling the movement of the endoscope manipulator based on the rotation angle. In some embodiments, the display area of the display device includes an exit area, and when the real-time gaze point position is located in the exit area and the gaze time exceeds a preset value, the computer device exits the endoscope adjustment mode and returns to the step of determining the manipulator adjustment mode according to the gaze point position. Figure 16 Shown is a schematic diagram of exiting the endoscope adjustment mode, wherein the black hexagonal dotted area in the figure represents the exit area, and the shape of the exit area can be any common shape and color, which is not further limited here.

[0139] When the robot arm adjustment mode is the tool adjustment mode, the computer device obtains the current captured image of the endoscope; and determines the target area in the current captured image according to the real-time gaze point position. Specifically, the computer device obtains the attention duration and the number of times the real-time gaze point position is focused, and determines the real-time area centered on the real-time gaze point position according to the attention duration and the number of times the real-time gaze point position is focused. Figure 17 The figure shows a schematic diagram of determining the real-time area corresponding to the real-time gaze point position. The longer the attention time, the larger the area of the real-time area, and the more times the attention is paid, the larger the area of the real-time area. Within the preset time, when the distance between the real-time gaze point positions at any two moments is within the preset area, the real-time area corresponding to the real-time gaze point position in the current captured image is determined as the target area. Figure 18 Shown is a schematic diagram for determining the target area.

[0140] The computer device receives the robot arm control instruction; when the robot arm control instruction is a zoom instruction, the target area is zoomed in and the zoomed target area is displayed on the display device; the tool robot arm is controlled to move according to the zoomed target area; when the robot arm control instruction is not a zoom instruction, the tool robot arm is controlled to move according to the target area. Figure 19 The figure shows a flow chart of controlling the movement of the tool manipulator according to the target area. Specifically, the movement of the tool manipulator is controlled according to the target area. Specifically, according to the position of the target area in the current captured image, the position of the tool manipulator's end device in the robot coordinate system is obtained by the conversion relationship between the display coordinate system and the robot coordinate system; according to the position of the end device, the twisting amount of each joint of the tool manipulator is determined by inverse kinematics; and the tool manipulator is controlled to move according to the twisting amount of each joint. Figure 20 The figure shows the process flow of the robot arm end control. Figure 21 The figure shows a schematic diagram of the robot arm returning to its initial position. The computer device determines whether to return to the initial position based on the real-time gaze point position. If the real-time gaze point position is in the "yes" position, the computer device controls the tool robot arm to move to the initial position. If the real-time gaze point position is in the "no" position, the computer device controls the tool robot arm to stop moving, exits the line of sight interaction system in the robot arm control system, and returns the robot arm to manual control.

[0141] The above-mentioned robotic arm control method collects real-time eye video of the target object looking at the display device, and the display device is used to display the real-time captured image of the endoscope on the area to be tested. Since the real-time gaze point position in the display device is obtained by eye tracking the real-time eye video of the target object, robotic arm control based on the real-time gaze point position is beneficial to improving the efficiency of robotic arm control; according to the real-time gaze point position, the robotic arm adjustment mode is determined. In the endoscope adjustment mode, the method of controlling the movement of the endoscope robotic arm according to the real-time gaze point position is beneficial to improving the control efficiency of the robotic arm; in the tool adjustment mode, the target object can update and adjust the angle of the endoscope in real time according to the actual situation observed by the endoscope, and automatically enter the tool adjustment mode when reaching the appropriate preset angle, which is beneficial to improving the control efficiency of the robotic arm; in the tool adjustment mode, the target area can be determined in the current captured image of the endoscope according to the real-time gaze point position, and then the movement of the tool robotic arm can be controlled. The determination of the robotic arm adjustment mode, the control of the endoscope robotic arm and the control of the tool robotic arm can be completed by the same target object by moving the eye line of sight, thereby improving the control efficiency of the robotic arm.

[0142] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0143] Based on the same inventive concept, embodiments of the present application further provide a robotic arm control device for implementing the aforementioned robotic arm control method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more robotic arm control device embodiments provided below can be found in the aforementioned limitations of the robotic arm control method and will not be further elaborated here.

[0144] In one embodiment, Figure 22 As shown, a robotic arm control device 100 is provided, comprising: an acquisition module 110, an eye tracking module 120, a mode determination module 130, a first control module 140 and a second control module 150, wherein:

[0145] The acquisition module 110 is used to acquire real-time eye video of the target object looking at the display device, and the display device is used to display the real-time acquired image of the endoscope on the area to be measured.

[0146] The eye tracking module 120 is used to determine the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video.

[0147] The mode determination module 130 is used to determine the adjustment mode of the robotic arm according to the real-time gaze point position.

[0148] The first control module 140 is used to control the movement of the endoscope manipulator according to the real-time gaze point position when the manipulator adjustment mode is the endoscope adjustment mode, until the endoscope angle reaches a preset angle, and update the manipulator adjustment mode to the tool adjustment mode.

[0149] The second control module 150 is used to obtain the current captured image of the endoscope when the robot arm adjustment mode is the tool adjustment mode; determine the target area in the current captured image according to the real-time gaze point position; and control the movement of the tool robot arm according to the target area.

[0150] The above-mentioned robotic arm control device collects real-time eye video of the target object looking at the display device, and the display device is used to display the real-time captured image of the endoscope on the area to be tested. Since the real-time gaze point position in the display device is obtained by eye tracking the real-time eye video of the target object, robotic arm control based on the real-time gaze point position is beneficial to improving the efficiency of robotic arm control; according to the real-time gaze point position, the robotic arm adjustment mode is determined. In the endoscope adjustment mode, the method of controlling the movement of the endoscope robotic arm according to the real-time gaze point position is used. The target object can update and adjust the angle of the endoscope in real time according to the actual situation observed by the endoscope, and automatically enter the tool adjustment mode when reaching the appropriate preset angle, which is beneficial to improving the control efficiency of the robotic arm; in the tool adjustment mode, the target area can be determined in the current captured image of the endoscope according to the position of the real-time gaze point, and then the movement of the tool robotic arm can be controlled. The determination of the robotic arm adjustment mode, the control of the endoscope robotic arm and the control of the tool robotic arm can be completed by the same target object by moving the eye line of sight, thereby improving the control efficiency of the robotic arm.

[0151] In one embodiment, in terms of controlling the movement of the endoscope robotic arm according to the real-time gaze point position, the first control module 140 is also used to: determine whether the real-time gaze point position is located within the endoscope adjustment area of the display device; when the real-time gaze point position is located within the endoscope adjustment area, determine the rotation angle according to the position of the endoscope adjustment area in the display device; and control the movement of the endoscope robotic arm based on the rotation angle.

[0152] In one embodiment, in terms of determining the target area in the current captured image based on the real-time gaze point position, the second control module 150 is further used to: obtain the attention duration and the number of times the real-time gaze point position is paid attention to; determine the real-time area centered on the real-time gaze point position based on the attention duration and the number of times it is paid attention to; within the preset time length, when the distance between the real-time gaze point positions at any two moments is within the preset area, the real-time area corresponding to the real-time gaze point position in the current captured image is determined as the target area.

[0153] In one embodiment, in terms of controlling the movement of the tool manipulator arm according to the target area, the second control module 150 is also used to: obtain the instrument end position of the tool manipulator arm in the robot coordinate system according to the position of the target area in the current acquired image through the conversion relationship between the display coordinate system and the robot coordinate system; determine the twisting amount of each joint of the tool manipulator arm through inverse kinematics according to the instrument end position; and control the tool manipulator arm to move according to the twisting amount of each joint.

[0154] In one embodiment, in terms of determining the real-time gaze point position of the target object in the display device through eye tracking based on the real-time eye video, the eye tracking module 120 is further used to: obtain the pupil position and Purkinje's spot position through pupil identification based on the real-time eye video; and determine the real-time gaze point position of the target object in the display device through a calibration mapping function based on the pupil position and the Purkinje's spot position.

[0155] In one embodiment, in terms of controlling the movement of the tool robot arm according to the target area, the second control module 150 is also used to: receive a robot arm control instruction; when the robot arm control instruction is a zoom instruction, enlarge the target area and display the enlarged target area on the display device; control the movement of the tool robot arm according to the enlarged target area; when the robot arm control instruction is not a zoom instruction, control the movement of the tool robot arm according to the target area.

[0156] Each module in the aforementioned robotic arm control device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0157] In one embodiment, a robotic arm control system is provided, the system comprising: a robotic arm, an image acquisition device, an endoscope, a display device, and a computer device. The robotic arm comprises an endoscope robotic arm and a tool robotic arm; the image acquisition device is used to acquire a real-time eye video of a target object looking at a display device; the endoscope is mounted at the end of the endoscope robotic arm and is used to acquire a real-time image of the area to be measured; the display device is used to display the real-time image of the area to be measured acquired by the endoscope; the computer device, which may be a terminal, may have an internal structure as shown in FIG. Figure 23As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for controlling a robotic arm is implemented.

[0158] Those skilled in the art will understand that Figure 23 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0159] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0160] A real-time eye video of the target object gazing at a display device is captured, and the display device is used to display the real-time captured image of the endoscope on the area to be measured; based on the real-time eye video, the real-time gaze point position of the target object in the display device is determined through eye tracking; based on the real-time gaze point position, the robot arm adjustment mode is determined; when the robot arm adjustment mode is the endoscope adjustment mode, the movement of the endoscope robot arm is controlled according to the real-time gaze point position until the endoscope angle reaches a preset angle, and the robot arm adjustment mode is updated to the tool adjustment mode; when the robot arm adjustment mode is the tool adjustment mode, the current captured image of the endoscope is obtained; based on the real-time gaze point position, the target area is determined in the current captured image; based on the target area, the movement of the tool robot arm is controlled.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] Determine whether the real-time gaze point position is within the endoscope adjustment area of the display device; if the real-time gaze point position is within the endoscope adjustment area, determine the rotation angle according to the position of the endoscope adjustment area in the display device; and control the movement of the endoscope robotic arm based on the rotation angle.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] Obtain the duration and number of times the real-time gaze point position is being watched; determine a real-time area centered on the real-time gaze point position based on the duration and number of times the real-time gaze point position is being watched; within a preset time period, when the distance between the real-time gaze point positions at any two moments is within the preset area, determine the real-time area corresponding to the real-time gaze point position in the current captured image as the target area.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] According to the position of the target area in the current acquired image, the position of the end device of the tool manipulator in the robot coordinate system is obtained through the conversion relationship between the display coordinate system and the robot coordinate system; according to the position of the end device, the twisting amount of each joint of the tool manipulator is determined through inverse kinematics; and the tool manipulator is controlled to move according to the twisting amount of each joint.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] According to the real-time eye video, the pupil position and Purkinje spot position are obtained through pupil recognition; according to the pupil position and Purkinje spot position, the real-time gaze point position of the target object in the display device is determined through a calibration mapping function.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Receive a robot arm control instruction; when the robot arm control instruction is a zoom instruction, enlarge the target area and display the enlarged target area on a display device; control the movement of the tool robot arm according to the enlarged target area; when the robot arm control instruction is not a zoom instruction, control the movement of the tool robot arm according to the target area.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0172] A real-time eye video of the target object gazing at a display device is captured, and the display device is used to display the real-time captured image of the endoscope on the area to be measured; based on the real-time eye video, the real-time gaze point position of the target object in the display device is determined through eye tracking; based on the real-time gaze point position, the robot arm adjustment mode is determined; when the robot arm adjustment mode is the endoscope adjustment mode, the movement of the endoscope robot arm is controlled according to the real-time gaze point position until the endoscope angle reaches a preset angle, and the robot arm adjustment mode is updated to the tool adjustment mode; when the robot arm adjustment mode is the tool adjustment mode, the current captured image of the endoscope is obtained; based on the real-time gaze point position, the target area is determined in the current captured image; based on the target area, the movement of the tool robot arm is controlled.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0174] Determine whether the real-time gaze point position is within the endoscope adjustment area of the display device; if the real-time gaze point position is within the endoscope adjustment area, determine the rotation angle according to the position of the endoscope adjustment area in the display device; and control the movement of the endoscope robotic arm based on the rotation angle.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0176] Obtain the duration and number of times the real-time gaze point position is being watched; determine a real-time area centered on the real-time gaze point position based on the duration and number of times the real-time gaze point position is being watched; within a preset time period, when the distance between the real-time gaze point positions at any two moments is within the preset area, determine the real-time area corresponding to the real-time gaze point position in the current captured image as the target area.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0178] According to the position of the target area in the current acquired image, the position of the end device of the tool manipulator in the robot coordinate system is obtained through the conversion relationship between the display coordinate system and the robot coordinate system; according to the position of the end device, the twisting amount of each joint of the tool manipulator is determined through inverse kinematics; and the tool manipulator is controlled to move according to the twisting amount of each joint.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0180] According to the real-time eye video, the pupil position and Purkinje spot position are obtained through pupil recognition; according to the pupil position and Purkinje spot position, the real-time gaze point position of the target object in the display device is determined through a calibration mapping function.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0182] Receive a robot arm control instruction; when the robot arm control instruction is a zoom instruction, enlarge the target area and display the enlarged target area on a display device; control the movement of the tool robot arm according to the enlarged target area; when the robot arm control instruction is not a zoom instruction, control the movement of the tool robot arm according to the target area.

[0183] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0184] A real-time eye video of the target object gazing at a display device is captured, and the display device is used to display the real-time captured image of the endoscope on the area to be measured; based on the real-time eye video, the real-time gaze point position of the target object in the display device is determined through eye tracking; based on the real-time gaze point position, the robot arm adjustment mode is determined; when the robot arm adjustment mode is the endoscope adjustment mode, the movement of the endoscope robot arm is controlled according to the real-time gaze point position until the endoscope angle reaches a preset angle, and the robot arm adjustment mode is updated to the tool adjustment mode; when the robot arm adjustment mode is the tool adjustment mode, the current captured image of the endoscope is obtained; based on the real-time gaze point position, the target area is determined in the current captured image; based on the target area, the movement of the tool robot arm is controlled.

[0185] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0186] Determine whether the real-time gaze point position is within the endoscope adjustment area of the display device; if the real-time gaze point position is within the endoscope adjustment area, determine the rotation angle according to the position of the endoscope adjustment area in the display device; and control the movement of the endoscope robotic arm based on the rotation angle.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0188] Obtain the duration and number of times the real-time gaze point position is being watched; determine a real-time area centered on the real-time gaze point position based on the duration and number of times the real-time gaze point position is being watched; within a preset time period, when the distance between the real-time gaze point positions at any two moments is within the preset area, determine the real-time area corresponding to the real-time gaze point position in the current captured image as the target area.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0190] According to the position of the target area in the current acquired image, the position of the end device of the tool manipulator in the robot coordinate system is obtained through the conversion relationship between the display coordinate system and the robot coordinate system; according to the position of the end device, the twisting amount of each joint of the tool manipulator is determined through inverse kinematics; and the tool manipulator is controlled to move according to the twisting amount of each joint.

[0191] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0192] According to the real-time eye video, the pupil position and Purkinje spot position are obtained through pupil recognition; according to the pupil position and Purkinje spot position, the real-time gaze point position of the target object in the display device is determined through a calibration mapping function.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0194] Receive a robot arm control instruction; when the robot arm control instruction is a zoom instruction, enlarge the target area and display the enlarged target area on a display device; control the movement of the tool robot arm according to the enlarged target area; when the robot arm control instruction is not a zoom instruction, control the movement of the tool robot arm according to the target area.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0196] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0197] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0198] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for controlling a robotic arm, characterized in that: The method comprises: Collecting real-time eye video of the target subject looking at a display device, wherein the display device is used to display a real-time image collected by the endoscope of the area to be measured; Determining the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video; Determining a robotic arm adjustment mode according to the real-time gaze point position; In the case where the manipulator adjustment mode is the endoscope adjustment mode, the endoscope manipulator is controlled to move according to the real-time gaze point position until the endoscope angle reaches a preset angle, and the manipulator adjustment mode is updated to the tool adjustment mode; When the manipulator adjustment mode is the tool adjustment mode, a current captured image of the endoscope is acquired; a target area is determined in the current captured image according to the real-time gaze point position; and a tool manipulator movement is controlled according to the target area; The determining of the target area in the current collected image according to the real-time gaze point position includes: Obtaining the attention duration and the number of times the real-time gaze point position is paid attention to; Determining a real-time area centered on the real-time gaze point position according to the attention duration and the number of attention times; When the distance between the real-time gaze point positions at any two moments within a preset time period is within a preset area, the real-time area corresponding to the real-time gaze point positions in the current captured image is determined as the target area.

2. The method according to claim 1, characterized in that The controlling the movement of the endoscope robotic arm according to the real-time gaze point position includes: determining whether the real-time gaze point position is within an endoscope adjustment area of the display device; In a case where the real-time gaze point position is located within the endoscope adjustment area, a rotation angle is determined according to the position of the endoscope adjustment area in the display device; and based on the rotation angle, the movement of the endoscope robotic arm is controlled.

3. The method according to claim 1, characterized in that The controlling the movement of the tool manipulator according to the target area includes: According to the position of the target area in the current acquired image, the position of the end of the tool manipulator in the robot coordinate system is obtained by converting the display coordinate system and the robot coordinate system; Determining the twisting amount of each joint of the tool manipulator arm through inverse kinematics according to the position of the end of the instrument; The tool manipulator is controlled to move according to the twisting amount of each joint.

4. The method according to any one of claims 1 to 3, characterized in that Determining the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video includes: Obtaining the pupil position and the Purkinje spot position by pupil recognition according to the real-time eye video; According to the pupil position and the Purkinje spot position, a real-time gaze point position of the target object in the display device is determined by calibrating a mapping function.

5. The method according to claim 1, characterized in that The controlling the movement of the tool manipulator according to the target area includes: Receive robotic arm control instructions; In the case where the robot arm control instruction is a zoom instruction, the target area is zoomed in and the zoomed target area is displayed on the display device; and the tool robot arm is controlled to move according to the zoomed target area; When the robot arm control instruction is not the amplification instruction, the tool robot arm is controlled to move according to the target area.

6. The method according to claim 1, wherein The determining of the robot arm adjustment mode according to the real-time gaze point position includes: When the real-time gaze point is located at the position of the endoscope adjustment mode, determining that the robot arm adjustment mode is the endoscope adjustment mode; When the real-time gaze point position is located at the position where the tool adjustment mode is located, the robot arm adjustment mode is determined to be the tool adjustment mode.

7. A robotic arm control device, characterized in that: The device comprises: An acquisition module is used to acquire real-time eye video of the target object looking at a display device, and the display device is used to display the real-time acquired image of the endoscope on the area to be measured; An eye tracking module, configured to determine the real-time gaze point position of the target object on the display device through eye tracking based on the real-time eye video; A mode determination module, configured to determine a robotic arm adjustment mode based on the real-time gaze point position; a first control module, configured to, when the manipulator adjustment mode is the endoscope adjustment mode, control the movement of the endoscope manipulator according to the real-time gaze point position until the endoscope angle reaches a preset angle, and update the manipulator adjustment mode to the tool adjustment mode; a second control module, configured to, when the manipulator adjustment mode is the tool adjustment mode, obtain a current captured image of the endoscope; determine a target area in the current captured image according to the real-time gaze point position; and control the movement of the tool manipulator according to the target area; The second control module is also used to obtain the attention duration and the number of times the real-time gaze point position is paid attention to; determine the real-time area centered on the real-time gaze point position based on the attention duration and the number of times the real-time gaze point position is paid attention to; within the preset time length, when the distance between the real-time gaze point positions at any two moments is within the preset area, the real-time area corresponding to the real-time gaze point position in the currently captured image is determined as the target area.

8. A robotic arm control system, characterized in that: The system comprises: Robotic arms, including endoscope robotic arms and tool robotic arms; An image acquisition device, used to acquire real-time eye video of a target object looking at a display device; An endoscope, which is mounted at the end of the endoscope robotic arm and is used to collect real-time images of the area to be measured; A display device, used to display the real-time image captured by the endoscope of the area to be measured; A computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images