Control method based on mixed reality, mixed reality equipment and surgical robot

Abstract

This specification discloses a manipulation method based on mixed reality, a mixed reality device, and a surgical robot. By rendering a virtual console and projecting it onto a mixed reality device worn by a user performing surgery to display it to the user, and monitoring the user's gesture operation on the virtual console, the controller identifier in the virtual console operated by the user is determined, and according to the controller identifier and gesture operation, a control instruction is sent to the surgical robot, so that the surgical robot operates according to the control instruction to perform the surgery. The virtual console rendered by the mixed reality device can be manipulated by a non-contact method with the mixed reality device as a control device to achieve control of the surgical robot, and the control of the virtual console will not be limited by the sterile membrane and inflexible, and there will be no situation where the sterile membrane is damaged due to manual operation, thereby ensuring the safety of the surgical process.

Description

Technical Field

[0001] This specification relates to the field of medical technology, and in particular to a control method based on mixed reality, a mixed reality device, and a surgical robot. Background Art

[0002] At present, surgical robots have been widely used in the medical field. In surgical scenarios, since surgical robots perform surgical operations (such as suturing, nailing, etc.) more stably than manual operations and can reduce the workload of doctors, the application of surgical robots in surgery has become a hot topic.

[0003] In traditional surgical scenarios, a surgical robot console needs to be configured in the operating room. The doctor manually operates the console to control the movement of the surgical robot and perform related surgical operations on the patient. Since the operation needs to be performed in a sterile environment, in order to reduce the risk of infection for the patient, the console is usually covered with a sterile film. The doctor controls the surgical robot to perform the operation by operating the console covered with a sterile film.

[0004] However, the sensitivity of the console covered with a sterile film will be reduced, especially when the console includes controllers such as joysticks that need to be moved, the sterile film will restrict the movement of the controller, so that the doctor's control over the surgical robot is limited. In addition, when the doctor manually operates the console covered with a sterile film, if the sterile film is damaged, the console will be exposed, making the sterile film ineffective and increasing the risk of infection for the patient. Summary of the invention

[0005] This specification provides a control method based on mixed reality, a mixed reality device and a surgical robot to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a control method based on mixed reality, including:

[0008] Rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery to display it to the user;

[0009] Determining, according to the monitored gesture operation of the user on the virtual console, a controller identifier in the virtual console operated by the user;

[0010] Sending a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction;

[0011] The operating result of the surgical robot according to the control instruction is received and displayed to the user.

[0012] Optionally, the mixed reality device is provided with a sensor;

[0013] Before determining the controller identifier in the virtual console operated by the user according to the monitored gesture operation of the user on the virtual console, the method further includes:

[0014] Monitoring the hand postures of the user in several frames by the sensor;

[0015] The user's gesture operation on the virtual console is determined according to the monitored hand postures of the plurality of frames and the posture of the virtual console.

[0016] Optionally, the method further comprises:

[0017] receiving a surgical plan sent by the surgical robot;

[0018] Verifying whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction;

[0019] If yes, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed;

[0020] If not, continue to identify the user's operation on the virtual console until the surgical plan is completed.

[0021] Optionally, the method further comprises:

[0022] receiving a verification result of the control instruction by the surgical robot;

[0023] When the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device, where the prompt message is used to prompt that the control instruction is not executed.

[0024] This specification provides a mixed reality device, which is used to control a surgical robot. A user performing surgery wears the mixed reality device, wherein:

[0025] The mixed reality device is used to render a virtual console and display it to the user; determine the controller identifier in the virtual console operated by the user based on the monitored gesture operation of the user on the virtual console; send a control instruction to the surgical robot based on the controller identifier and the gesture operation; receive the operation result of the surgical robot according to the control instruction, and display it to the user.

[0026] Optionally, the mixed reality device is provided with a sensor;

[0027] The mixed reality device is also used to monitor the user's hand postures in several frames through the sensor, and determine the user's gesture operation on the virtual console based on the monitored hand postures in several frames and the posture of the virtual console.

[0028] Optionally, the mixed reality device is also used to receive the surgical plan sent by the surgical robot, and verify whether the surgical plan is completed based on the operating results of the surgical robot according to the control instructions; when the surgical plan is completed, stop monitoring the user's gesture operations on the virtual console, and prompt the user that the surgical plan is completed; when the surgical plan is not completed, continue to identify the user's operations on the virtual console until the surgical plan is completed.

[0029] Optionally, the mixed reality device is further used to receive a verification result of the surgical robot on the control instruction, and when the verification result is unsafe, display a prompt message to the user, wherein the prompt message is used to prompt that the control instruction has not been executed.

[0030] The present specification provides a surgical robot, which is used to receive control instructions sent by a mixed reality device worn by a user performing an operation, operate according to the control instructions, and send the operating results to the mixed reality device; the control instructions are determined by the mixed reality device based on the user's gesture operation on a rendered virtual console and a controller identifier in the virtual console operated by the user.

[0031] Optionally, the surgical robot is also used to determine a preset surgical plan and send the surgical plan to the mixed reality device.

[0032] Optionally, the surgical robot is further used to verify the received control instruction at least according to the surgical plan, and send the verification result of the control instruction to the mixed reality device.

[0033] This specification provides a control device based on mixed reality, including:

[0034] A receiving module, used for rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery, so as to be displayed to the user;

[0035] a rendering module, configured to determine, according to the monitored gesture operation of the user on the virtual console, a controller identifier in the virtual console operated by the user;

[0036] an identification module, configured to send a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction;

[0037] The control module is used to receive the operation result of the surgical robot according to the control instruction and display it to the user.

[0038] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned control method based on mixed reality is implemented.

[0039] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned control method based on mixed reality when executing the program.

[0040] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0041] In the mixed reality-based control method provided in the present specification, a virtual console is rendered and projected onto a mixed reality device worn by a user performing the operation to display it to the user, and the user's gesture operations on the virtual console are monitored to determine the controller identifier in the virtual console operated by the user. Based on the controller identifier and the gesture operation, control instructions are sent to the surgical robot so that the surgical robot operates according to the control instructions to perform the operation.

[0042] It can be seen from the above method that this method can control the surgical robot by manipulating the virtual console rendered by the mixed reality device without contact with the mixed reality device as a control device. The control of the virtual console will not be restricted by the sterile membrane and will not be inflexible, nor will the sterile membrane be damaged due to manual operation, thereby ensuring the safety of the surgical process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0044] Figure 1 A schematic diagram of a control method based on mixed reality in this specification;

[0045] Figure 2 A schematic diagram of a mixed reality-based surgical scene provided in this manual;

[0046] Figure 3 A schematic diagram of the display content of a mixed reality device provided in this manual;

[0047] Figure 4 A schematic diagram of the interaction process between a mixed reality device and a surgical robot provided in this specification;

[0048] Figure 5 A schematic diagram of a mixed reality-based control device provided in this specification;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this manual. DETAILED DESCRIPTION

[0050] In traditional surgical scenarios, since the operation needs to be performed in a sterile environment, in order to reduce the patient's risk of infection, when the doctor operates the console in the operating room to control the surgical robot, the console and the surgical robot need to be sterilized, for example, by disinfecting the surgical robot and the console with disinfectants, or covering them with sterile film.

[0051] A surgical robot is a device that needs to perform related operations corresponding to the surgery on the patient's skin surface or inside the body. It is usually a device that is easier to clean and disinfect than a console. The console needs to integrate multiple functions, and the size of the console adapted to the surgical robot used in different surgical scenarios is also different, and the structure is also different. For some consoles, their surfaces are easier to clean with disinfectants, but the internal parts such as gaps and inner walls are difficult to be completely sterilized. In addition, the patient's body fluids, medicines, etc. during the operation are also easy to contaminate the console, making the subsequent cleaning of the console more difficult, and the cleaning intensity needs to be increased. The cleaning agent may cause corrosion to the console. As the number of cleanings accumulates, the corrosion will become more serious, and even cause the controller in the console to partially fail, and the corresponding maintenance cost of the console will also increase. Therefore, the method of covering the console with a sterile film is usually used to sterilize the console.

[0052] However, the sensitivity of the console covered with a sterile film will be reduced, especially when the console includes controllers such as joysticks that need to be moved, the sterile film will limit the movement of the controller to a certain extent, limiting the doctor's control over the surgical robot. In addition, when the doctor manually operates the console covered with a sterile film, if the sterile film is damaged, the console will be exposed, making the sterile film ineffective, increasing the risk of infection for the patient, and exposing the patient to a dangerous environment.

[0053] In another surgical scenario, the doctor and the control console are both in a space far away from the operating room. The doctor remotely controls the surgical robot through the control console to operate on the patient's surgical site, and observes the patient or the surgical robot's operation on the surgical site through a transparent medium or monitoring with the operating room. However, in this method, since the doctor is not in the operating room, the line of sight and field of vision will be limited by factors such as occlusion and angle, making it impossible to observe the patient and the execution of the operation from the best perspective, and even produce visual illusions due to angle problems, resulting in errors in the operation of the surgical robot, increasing the risk of the operation, and even leading to the failure of the operation.

[0054] In order to at least partially solve these problems, in the embodiments provided in this specification, the doctor and the console are both in the operating room, and the doctor can observe the patient and the operation of the surgical robot on the surgical site from any perspective. In addition, the console in this specification is a virtual console rendered by a mixed reality device, and the mixed reality device can specifically be a head-mounted mixed reality device, such as smart glasses integrated with mixed reality technology.

[0055] Mixed Reality (MR) devices are devices that can render virtual 3D scenes based on the real environment seen by the wearer and display them to the user, and can realize the interaction between reality and virtuality. For example, a mixed reality device can display a mixed scene of the real environment and the virtual environment to the wearer, and recognize the wearer's gestures, sounds and other information, and respond, such as: with the wearer's rotation gesture recognized, the virtual 3D object displayed to the wearer rotates with the wearer's gesture.

[0056] In the embodiments provided in this specification, users (such as doctors and other medical staff) can wear a mixed reality device connected to a surgical robot during surgery. And control of the surgical robot is achieved by operating the virtual console rendered by the mixed reality device. The mixed reality device can be covered with a sterile film, and the user only wears the mixed reality device as a control device. After wearing it, there is no need to operate and touch the mixed reality device during surgery, but to control the surgical robot by operating the virtual console. The console contains several controllers for controlling the surgical robot to achieve corresponding operations and functions. The user controls the surgical robot by operating the controller in the virtual console. Since the virtual console is virtual, it does not need to be sterilized, and the operation of the controller will not be restricted due to the covering of the console by the sterile film, that is, there will be no inflexible operation of the controller, and the sterile film will not be damaged due to manual operation of the console, increasing the risk of infection for patients.

[0057] It should be noted that in this specification, the surgical robot and the mixed reality device can communicate through a wired connection or a wireless connection, which can be set according to needs and this specification does not limit it.

[0058] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0059] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0060] Figure 1 This is a schematic diagram of a control method based on mixed reality in this specification, which specifically includes the following steps:

[0061] S100: Rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery, so as to be displayed to the user.

[0062] In this specification, the mixed reality-based manipulation method can be performed by a mixed reality device worn by a user performing surgery. The mixed reality device is a device used to control the surgical robot. The surgical robot is used to perform surgery on a patient under the control of the surgical robot by the user through the mixed reality device.

[0063] In one or more embodiments of the present specification, the mixed reality device may first render a virtual console, and project the virtual console onto the mixed reality device worn by the user performing the surgery, so as to display it to the user. Therefore, the mixed reality device may at least include a display device, and if the mixed reality device is a smart glasses, the display device may be a lens. The user may manipulate the controller in the virtual console seen through the mixed reality device to control the surgical robot.

[0064] The virtual console is a console rendered by the mixed reality device that includes several virtual controllers, each of which is used to implement at least one function or operation of the surgical robot. For example, the controller of the virtual console may include: a controller for controlling the up and down movement of the surgical robot, a controller for controlling the surgical robot to perform the corresponding cutting operation of the surgery, a controller for controlling the surgical robot to perform the corresponding nailing operation, suturing operation, puncture operation and other operations of the surgery, etc. Of course, the controllers included in the console mentioned in this specification are only examples and not limitations, and can be flexibly set according to the nature of the surgery and the needs of the surgery, and this specification does not limit it here.

[0065] In addition, the function, form and number of controllers included in the virtual console can be set as needed, and this specification does not limit this. The relevant parameters for rendering the virtual console can be pre-designed and stored in the mixed reality device according to the nature of the operation to be performed and the needs of the operation. When the mixed reality device is started, the mixed reality device can render the virtual console for executing it. That is, according to the operation to be performed, the virtual console can be pre-set to a fixed form, and according to the operation to be performed, the corresponding functions of each controller in the virtual console are assigned. For example, assuming that the operation to be performed is an operation to place nails in multiple bone positions of the patient, the surgical robot controlled by the mixed reality device needs to have the ability to move and place nails, and the corresponding virtual console may include a controller for controlling the movement of the surgical robot and a controller for controlling the surgical robot to perform the nail placement operation.

[0066] S102: Determine, according to the monitored gesture operation of the user on the virtual console, a controller identifier in the virtual console operated by the user.

[0067] In this specification, each controller in the virtual console has a unique controller identifier. The mixed reality device can monitor the user's gesture operation on the virtual console in real time, determine the controller identifier in the virtual console operated by the user, so as to determine the control instructions issued to the surgical robot in the subsequent steps, that is, determine the operation to be controlled by the surgical robot.

[0068] S104: Sending a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction.

[0069] In one or more embodiments of the present specification, after determining the controller identifier of the user operation and the user's gesture operation, the mixed reality device can send a control instruction to the surgical robot based on the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction and performs the target operation indicated by the control instruction.

[0070] The control instruction carries at least an operation parameter, which is used to control the surgical robot to perform the target operation indicated by the control instruction. The target operation is the operation corresponding to the function of the controller to which the controller identifier belongs, triggered by the user's gesture operation.

[0071] S106: Receive the operating result of the surgical robot according to the control instruction, and display it to the user.

[0072] In one or more embodiments of the present specification, the surgical robot can operate according to the control instruction sent by the mixed reality device, that is, execute the target operation indicated by the control instruction, and send the operation result, that is, the execution result of the target operation, to the mixed reality device.

[0073] Therefore, in one or more embodiments of the present specification, the mixed reality device may also receive the operation result of the surgical robot according to the control instruction, and display the operation result to the user, so that the user can perform corresponding operations according to the operation result displayed by the mixed reality device.

[0074] Alternatively, after receiving the operation result, the mixed reality device may also convey the operation result to the user through voice notification.

[0075] The operation result includes execution success and execution failure. When the operation result is execution success, the user can continue to perform subsequent operations or end the operation. When the operation result is failure, the user can control the controller in the virtual console again by initiating gesture operations, or perform other operations, such as terminating the operation, checking the surgical robot, or performing other operations on the patient.

[0076] based on Figure 1 The mixed reality-based control method shown in the figure renders a virtual console and projects it onto a mixed reality device worn by a user performing the operation to display it to the user, monitors the user's gesture operations on the virtual console, determines the controller identifier in the virtual console operated by the user, and sends control instructions to the surgical robot based on the controller identifier and gesture operations, so that the surgical robot operates according to the control instructions to perform the operation.

[0077] It can be seen from the above method that this method can control the surgical robot by manipulating the virtual console rendered by the mixed reality device without contact with the mixed reality device as a control device. The control of the virtual console will not be restricted by the sterile membrane and will not be inflexible, nor will the sterile membrane be damaged due to manual operation, thereby ensuring the safety of the surgical process.

[0078] In addition, in one or more embodiments of this specification, since the mixed reality device needs to monitor the user's gesture operation, a sensor may be provided on the mixed reality device. The sensor may be provided as required, for example, it may be a visual sensor, an infrared sensor, etc., or other sensors, as long as it can recognize the user's gesture operation on the controller, and this specification does not limit it here.

[0079] Based on this, before determining the controller identifier in the virtual console operated by the user according to the monitored gesture operation of the user on the virtual console in step S102, the mixed reality device can also monitor the hand postures of the user in several frames through sensors, and determine the gesture operation of the user on the virtual console according to the monitored hand postures of several frames and the posture of the virtual console.

[0080] In addition, in one or more embodiments of the present specification, the surgical robot may also store a surgical plan for the operation to be performed, and send the surgical plan to the mixed reality device. The mixed reality device may also receive the surgical plan sent by the surgical robot, and after receiving the operating results of the surgical robot in response to the control instruction and the surgical plan, verify whether the surgical plan is completed based on the operating results of the received surgical robot in response to the control instruction. If so, stop monitoring the user's gesture operations on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed. Avoid performing operations outside the surgical plan or other risky operations on the patient due to monitoring of gesture operations that are mistakenly triggered by the user, thereby further ensuring the safety of the operation.

[0081] In one or more embodiments of the present specification, if the surgical plan is not completed, the mixed reality device may continue to recognize the user's operation on the virtual console until the surgical plan is completed.

[0082] Among them, the surgical plan may include the pre-planned specific operations to be performed on the patient when performing the operation to be performed, the tools, quantity, operation position, etc. of the operation, or may also include relevant data of the patient corresponding to the operation, such as a computed tomography (CT) image, etc. Taking suturing surgery as an example, the surgical plan of the operation may include suturing, the location of suturing, the method of suturing, etc. The surgical plan may include a whitelist and / or a blacklist of operations to be performed on the patient. The surgical robot may only execute control instructions corresponding to the operations in the whitelist sent by the mixed reality device, or execute control instructions corresponding to other operations other than the prohibited operations in the blacklist. The specific settings can be made as needed, and this manual does not limit it here.

[0083] In one or more embodiments of the present specification, after receiving the surgical plan, the mixed reality device can display the surgical plan to the user, so that the user can perform the surgery corresponding to the surgical plan on the patient according to the surgical plan, or refer to the surgical plan and perform the surgery in combination with the patient's condition during the operation.

[0084] In this specification, the surgical robot can also send the current state parameters of the surgical robot to the mixed reality device in real time when it is started or during the operation. The surgical robot can be configured with a chip encapsulated with a program for realizing the functions of the surgical robot, and the state parameters include at least: the running state of the program and the device state and device parameters of the surgical robot. For example, the running state may include: running failure, inoperable surgery, executable surgery, etc. The device state may include whether the surgical robot is in an executable state (i.e., whether it can respond to the control instructions of the mixed reality device and perform the target operation), whether it is faulty, etc. The device parameters include at least: the current posture of the surgical robot, the movable range of the surgical robot, the moving speed range of the surgical robot, etc.

[0085] Since it may be necessary to switch different surgical robots or components configured on the surgical robot during surgery to achieve different functions to perform different surgical operations on the patient. For example, the surgical robot may be composed of a body and an actuator, and the surgical robot is a device that can switch different actuators, that is, a device that can switch and dock different actuators on the body. Different actuators correspond to different functions, so that when the surgical robot docks with different actuators, the meaning of the instructions for controlling the surgical robot that can be triggered by the virtual console of the mixed reality device is different. Therefore, the device parameters of the surgical robot may also include the type of equipment (such as a scalpel, suture needle, puncture needle, C-arm, etc.). Accordingly, the mixed reality device may be configured with a correspondence between the switchable surgical robot and the function of the virtual console during surgery, so that when the surgical robot or actuator is switched, the mixed reality device can be based on the correspondence between the switched surgical robot and the function of the controller in the virtual console, and further update the meaning of the operation that can be triggered by the controller in the virtual console, so that the mixed reality device can determine the accurate control instructions according to the recognized user's gesture operation and controller identification.

[0086] In one or more embodiments of the present specification, the mixed reality device may also display the current status parameters of the surgical robot to the user or prompt the user through voice when receiving the current status parameters of the surgical robot sent by the surgical robot.

[0087] In one or more embodiments of the present specification, the surgical robot may send the surgical plan to the mixed reality device upon startup. In step S100, the virtual console rendered by the mixed reality device may be a virtual console adapted to the surgery to be performed and rendered based on the received surgical plan. Alternatively, the mixed reality device may store configuration parameters of virtual consoles adapted to a variety of surgeries. Then, when receiving the surgical plan, the mixed reality device may also determine a virtual console adapted to the surgical plan from a number of pre-configured virtual consoles and render the virtual console.

[0088] In addition, since the safety of the operation is a key point in the execution process of the operation, in one or more embodiments of the present specification, the surgical robot can verify the control instruction sent by the mixed reality device at least according to the operation plan to verify whether the control instruction is safe. And send the verification result to the mixed reality device. When the verification result is unsafe, the surgical robot does not perform the target operation specified by the control instruction. When the verification result is safe, the surgical robot performs the target operation specified by the control instruction.

[0089] In one or more embodiments of the present specification, when the surgical robot verifies whether the target operation indicated by the control instruction is safe according to the surgical plan, it can determine whether the target operation is within the range allowed by the surgical plan. If not, it is determined that the target operation indicated by the control instruction is unsafe; if so, it is determined that the target operation indicated by the control instruction is safe.

[0090] In one or more embodiments of the present specification, when the surgical robot verifies whether the target operation indicated by the control instruction is safe according to the surgical plan, it can determine whether the target operation is within the range prohibited by the surgical plan. If so, it is determined that the target operation indicated by the control instruction is unsafe; if not, it is determined that the target operation indicated by the control instruction is safe.

[0091] Specifically, taking the example that the surgery to be performed is a nail placement surgery, the surgical plan may include the target nail placement positions in the patient's surgical site where the nail placement operation can be performed and the total number of nails to be placed in the surgery to be performed. Assuming that the target operation is to place a nail at position A, the surgical robot may verify whether position A is a target nail placement position in a safe position according to the surgical plan when receiving the control instruction, and verify whether the number of nails placed on the patient as of now is not greater than the total number of nails in the surgical plan. If the verification results are all yes, it is determined that the target operation indicated by the control instruction is safe. Otherwise, it is determined that the target operation indicated by the control instruction is unsafe.

[0092] Since the surgical robot may need to perform operations on multiple positions during the operation, the surgical robot has the ability to move, and the movement can be overall movement or local movement. For example, the surgical robot can include a fixed base and a movable robotic arm.

[0093] Accordingly, when the surgical robot needs to be controlled to move, the user can manipulate the virtual console used to control the movement of the surgical robot, and the mixed reality device can send a control instruction to the surgical robot. The operation parameters carried by the control instruction at least include the operation position for performing the target operation, that is, the target position to which the surgical robot needs to be controlled to move.

[0094] The surgical robot can move to the operating position according to the control instruction.

[0095] In addition, when executing the target operation indicated by the control instruction, the surgical robot can also determine whether it has moved to the operation position. If so, it executes the target operation indicated by the control instruction; if not, it continues to move to the operation position.

[0096] Alternatively, the operating parameter may include the current position of the surgical robot and the operating position for performing the target operation. In one or more embodiments of the present specification, in order to ensure that the mixed reality device maintains a good synchronization relationship with the surgical robot, the surgical robot is also used to verify whether the current position is consistent with the current actual position of the surgical robot when receiving the control instruction. If so, it moves to the operating position, and if not, it sends an error message to the mixed reality device. Avoid situations where the mixed reality device fails to maintain good synchronization with the surgical robot due to unnoticed faults or errors during the execution of the operation, and the surgical robot has a wrong understanding of its current position, resulting in risks in the operation.

[0097] In one or more embodiments of the present specification, the mixed reality device can also receive error information sent by the surgical robot and prompt the user based on the error information.

[0098] Alternatively, since the operation to be performed on the patient during the operation may not be a single operation but a sequence of operations, for example, suturing the skin from position A to position B, the target operation indicated by the control instruction involves at least two target positions, i.e., at least two operation positions. Therefore, the operation parameter may also include at least the starting operation position and the target operation position of the surgical robot. Or the current position, starting operation position, and target operation position of the surgical robot.

[0099] In one or more embodiments of the present specification, the operation parameter may also include an operation speed. The surgical robot is further configured to move to the operation position at the operation speed, or move from the current position to the starting operation position, and then move from the starting operation position to the target operation position when executing the target operation indicated by the control instruction.

[0100] In one or more embodiments of the present specification, the operation parameter may further include an operation action and an operation speed. The surgical robot may also perform the target operation at the operation speed when performing the target operation indicated by the control instruction.

[0101] To summarize, the operation parameters may include: operation position, current position, starting operation position, target operation position, operation speed and a combination of one or more operation actions. Of course, they may also include others, such as operation force, operation amplitude, etc., which are not limited in this manual.

[0102] In one or more embodiments of this specification, the mixed reality device can also determine the operating parameters of the surgical robot according to the combination of one or more of the recognized operation mode, operation strength, operation frequency, operation amplitude, and operation speed of the user on the virtual console. Of course, it can also be determined in other ways, which are not limited in this specification.

[0103] Since it is difficult for the user to observe with the naked eye whether some operations of the surgical robot on the patient (such as subcutaneous operations) are completed or executed successfully, in one or more embodiments of the present specification, when the surgical robot executes the target operation indicated by the control instruction, it can also feedback the execution status of the target operation to the mixed reality device, and the mixed reality device can receive the execution status sent by the surgical robot and prompt the user. Among them, the execution status includes: executing and execution completed. Alternatively, the execution status may correspond to the target operation. For example, if the target operation is to move from position A to position B, then in the process of the surgical robot moving from position A to position B, before reaching position B, the execution status may be "moving". The execution status of the end of execution is used to prompt the user that the surgical robot will no longer continue to perform the target operation, and does not necessarily mean that the target operation is successfully executed.

[0104] In one or more embodiments of the present specification, the mixed reality device may also receive the verification result of the surgical robot on the control instruction. When the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device. The prompt message is used to prompt that the control instruction is not executed.

[0105] It should be noted that in one or more embodiments of the present specification, the target operation can be a real-time operation or a process operation that needs to be performed over a time interval. When the target operation is a real-time operation, the user can continuously control the controller through a number of gesture operations, and observe through the mixed reality device that the surgical robot continuously performs a number of target operations corresponding to the gesture operations. When the target operation is a process operation that needs to be performed over a time interval, the user can intermittently control the controller through gesture operations, and after observing through the mixed reality device that the surgical robot performs the target operation corresponding to the gesture operation, initiate a new gesture operation.

[0106] In one or more embodiments of the present specification, the user may also control the mixed reality device by voice to send control instructions to the surgical robot. The mixed reality device may also perform voice recognition on the voice data when collecting the user's voice data to determine the operation parameters contained in the voice data.

[0107] Among them, when performing voice recognition, existing voice recognition technology can be used, and this manual will not elaborate on it here.

[0108] Then, the mixed reality device can also prompt the user through voice prompts the execution result of the surgical robot on the control instruction, the state parameters of the surgical robot, the verification result of the surgical robot on the control instruction, etc. Of course, other contents involved in the surgical process can also be included, and this manual does not limit it here.

[0109] In one or more embodiments of the present specification, the mixed reality device may also monitor user-triggered operations for controlling the surgical robot in other ways, for example, by tracking the user's eye movements through sensors, etc.

[0110] In this specification, there is no restriction on the structure and composition of the mixed reality device and the surgical robot. For example, the surgical robot may include an actuator and a drive module, and the drive module may be used to control the actuator to perform the target operation indicated by the control instruction according to the operating parameters carried by the received control instruction. Since the surgical robot needs to communicate with the mixed reality device, the surgical robot may also include a communication module. Accordingly, the mixed reality device may also include a communication module for communicating with the surgical robot.

[0111] Figure 2A schematic diagram of a surgical scene based on mixed reality is provided in this specification. As shown in the figure, the system includes a mixed reality device D and a surgical robot K. The surgical robot includes an actuator B, a drive module B01, a communication module C, and a chip C01 encapsulated with a program that implements the function of the surgical robot. A represents the patient, G represents the surgical site, and H represents the bed, which is located in the sterile area. It can be seen that the mixed reality device D is capable of rendering a mixed control environment that includes at least a virtual console E, and the user F wearing the mixed reality device D can control the surgical robot K by controlling the virtual console E. ①, ②, and ③ respectively represent several positions of the actuator B during the movement process.

[0112] In one or more embodiments of the present specification, the content displayed to the user by the mixed reality device includes at least a virtual console. Of course, in addition to the virtual console, other content may also be displayed, such as Figure 3 As shown, of course, the specific content to be displayed can be set as needed, and this manual does not limit it here.

[0113] Figure 3 A schematic diagram of the display content of a mixed reality device provided in this specification. As shown in the figure, the field of view displayed to the user by the mixed reality device is shown. It can be seen that in addition to the real environment background seen by the user, the field of view also includes the CT image taken by the patient before the operation and the three-dimensional model of the patient's bones constructed based on the CT image. The user can refer to the displayed content and control the surgical robot to perform the operation through the mixed reality device. Among them, the dotted box is a virtual console rendered by the mixed reality device, and the virtual console contains several controllers.

[0114] This manual also provides the interaction process between the mixed reality device and the surgical robot. Figure 4 .

[0115] Figure 4 The following is a schematic diagram of the interaction process between a mixed reality device and a surgical robot provided in this specification. The interaction process includes:

[0116] S300: The mixed reality device renders a virtual console and displays the virtual console to a user wearing the mixed reality device.

[0117] S301: The surgical robot determines the preoperative plan and its own equipment parameters.

[0118] S302: The surgical robot sends the determined preoperative plan and device parameters to the mixed reality device.

[0119] S303: The mixed reality device monitors the user's gesture operation and determines the controller identifier of the user's operation.

[0120] S304: The mixed reality device sends a control instruction carrying an operation parameter to the surgical robot according to the gesture operation and the controller identifier, where the control instruction is used to instruct the surgical robot to perform a target operation indicated by the control instruction according to the operation parameter.

[0121] S305: The surgical robot verifies whether the received control instruction is safe at least according to the surgical plan. If so, execute steps S307 to S308 and S310; otherwise, execute S306.

[0122] S306: The surgical robot feeds back the verification result to the mixed reality device.

[0123] S307: The surgical robot feeds back the verification result to the mixed reality device.

[0124] S308: The surgical robot operates according to the operating parameters carried by the control instruction and performs the target operation indicated by the control instruction.

[0125] S309: The mixed reality device displays the received verification result to the user.

[0126] S310: The surgical robot feeds back the operating results to the mixed reality device.

[0127] S311: The mixed reality device displays the operation result to the user.

[0128] S312: The mixed reality device verifies whether the surgical plan has been executed based on the operation result. If so, step S313 is executed; if not, step S303 is repeated.

[0129] S313: Prompt the user that the surgical plan is finished, and stop monitoring the user's gesture operation.

[0130] It should be noted that, although the mixed reality device and the surgical robot may interact multiple times during the operation, in the above steps S300 to S313, when the steps performed by the mixed reality device and the surgical robot do not affect, interfere with, or contradict each other, the two can perform the corresponding steps simultaneously, or they can perform them one after another, and this specification does not limit this. The same is true for steps S100 to S108.

[0131] based on Figure 1 In addition to the mixed reality-based control method shown, this specification also provides a mixed reality device.

[0132] The mixed reality device is used to control the surgical robot, and the user performing the surgery wears the mixed reality device, wherein:

[0133] The mixed reality device is used to render a virtual console and display it to the user. According to the gesture operation of the user on the virtual console monitored by the user, the controller identifier in the virtual console operated by the user is determined. According to the controller identifier and the gesture operation, a control instruction is sent to the surgical robot, and the operation result of the surgical robot according to the control instruction is received, and the operation result is displayed to the user.

[0134] In one or more embodiments of the present specification, the mixed reality device may be provided with a sensor. The mixed reality device is also used to monitor the hand postures of the user in several frames through the sensor, and determine the user's gesture operation on the virtual console based on the monitored hand postures of the several frames and the position of the virtual console.

[0135] In addition, the mixed reality device is also used to receive the surgical plan sent by the surgical robot, and verify whether the surgical plan is completed based on the received results of the operation of the surgical robot according to the control instructions. When the surgical plan is completed, it stops monitoring the user's gesture operation on the virtual console and prompts the user that the surgical plan is completed.

[0136] In one or more embodiments of the present specification, the mixed reality device is also used to continue to recognize the user's operations on the virtual console when the surgical plan is not completed until the surgical plan is completed.

[0137] In one or more embodiments of the present specification, the mixed reality device is further used to receive the verification result of the control instruction by the surgical robot, and when the verification result is unsafe, display a prompt message to the user, re-monitor the user's gesture operation on the virtual console, and send a control instruction to the surgical robot based on the re-identified user's gesture operation on the virtual console and the controller identifier in the virtual console operated by the user. The prompt message is used to indicate that the control instruction has not been executed.

[0138] The present specification also provides a surgical robot, which is used to receive control instructions sent by a mixed reality device worn by a user performing an operation, operate according to the control instructions, and send the operating results to the mixed reality device. The control instructions are determined by the mixed reality device based on the user's gesture operation on the rendered virtual console and the controller identifier in the virtual console operated by the user.

[0139] In this specification, the surgical robot is also used to determine a preset surgical plan and send the surgical plan to a mixed reality device.

[0140] In one or more embodiments of the present specification, the surgical robot is also used to verify the received control instructions at least according to the surgical plan, and send the verification results of the control instructions to the mixed reality device.

[0141] It should be noted that the corresponding contents of the mixed reality-based control method, mixed reality device and surgical robot provided in the above-mentioned specification can be referenced to each other.

[0142] based on Figure 1 The present specification also provides a corresponding control device based on mixed reality, such as Figure 5 shown.

[0143] Figure 5 A schematic diagram of a mixed reality based control device provided in this specification, the device includes:

[0144] A rendering module 200, used for rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery, so as to be displayed to the user;

[0145] A monitoring module 201, configured to determine a controller identifier in the virtual console operated by the user according to the monitored gesture operation of the user on the virtual console;

[0146] A control module 202, configured to send a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction;

[0147] The display module 203 is used to receive the operation result of the surgical robot according to the control instruction and display it to the user.

[0148] Optionally, a sensor is provided on the mixed reality device, and the monitoring module 201 is further used to monitor the hand postures of several frames of the user through the sensor, and determine the gesture operation of the user on the virtual console according to the monitored hand postures of the several frames and the posture of the virtual console.

[0149] The device also includes:

[0150] The receiving module 204 is used to receive the surgical plan sent by the surgical robot, and verify whether the surgical plan is completed based on the operating result of the surgical robot according to the control instruction. If so, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed. If not, continue to identify the user's operation on the virtual console until the surgical plan is completed.

[0151] The device also includes:

[0152] The prompt module 205 is used to receive the verification result of the surgical robot on the control instruction. When the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device, and the prompt message is used to prompt that the control instruction has not been executed.

[0153] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A control method based on mixed reality is provided.

[0154] This manual also provides Figure 6 The structural diagram of the electronic device shown in FIG. Figure 6 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 A control method based on mixed reality is provided.

[0155] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0156] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0157] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0158] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0159] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0160] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0166] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0167] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0168] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0170] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A control method based on mixed reality, characterized in that: include: Rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery to display it to the user; Determining, according to the monitored gesture operation of the user on the virtual console, a controller identifier in the virtual console operated by the user; Sending a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction; receiving the operating result of the surgical robot according to the control instruction, and displaying it to the user; It also includes: receiving a surgical plan sent by the surgical robot; Verifying whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction; If yes, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed; If not, continue to identify the user's operation on the virtual console until the surgical plan is completed; And also includes: after sending the control instruction to the surgical robot, receiving the verification result of the control instruction by the surgical robot; When the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device, where the prompt message is used to prompt that the control instruction is not executed.

2. The method according to claim 1, characterized in that The mixed reality device is provided with a sensor; Before determining the controller identifier in the virtual console operated by the user according to the monitored gesture operation of the user on the virtual console, the method further includes: Monitoring the hand postures of the user in several frames by the sensor; The user's gesture operation on the virtual console is determined according to the monitored hand postures of the plurality of frames and the posture of the virtual console.

3. A mixed reality device, characterized in that: The mixed reality device is used to control the surgical robot, and the user performing the surgery wears the mixed reality device, wherein: The mixed reality device is used to perform a mixed reality control method, which includes: rendering a virtual console and projecting it onto the mixed reality device worn by a user performing surgery to display it to the user; determining a controller identifier in the virtual console operated by the user based on a monitored gesture operation of the user on the virtual console; sending a control instruction to a surgical robot based on the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction; receiving an operation result of the surgical robot according to the control instruction, and displaying it to the user; The method further includes: receiving a surgical plan sent by the surgical robot; and verifying whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction; If yes, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed; If not, continue to identify the user's operation on the virtual console until the surgical plan is completed; And it also includes: after sending a control instruction to the surgical robot, receiving the verification result of the control instruction by the surgical robot; when the verification result is unsafe, displaying prompt information to the user through the mixed reality device, and the prompt information is used to prompt that the control instruction is not executed.

4. The mixed reality device according to claim 3, wherein: The mixed reality device is provided with a sensor; The mixed reality device is also used to monitor the user's hand postures in several frames through the sensor, and determine the user's gesture operation on the virtual console based on the monitored hand postures in several frames and the posture of the virtual console.

5. A surgical robot, characterized in that: The surgical robot is used to perform a mixed reality control method, which includes: rendering a virtual console and projecting it onto a mixed reality device worn by a user performing surgery to display it to the user; determining a controller identifier in the virtual console operated by the user based on a monitored gesture operation of the user on the virtual console; sending a control instruction to the surgical robot based on the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction; receiving an operation result of the surgical robot according to the control instruction, and displaying it to the user; The method further includes: receiving a surgical plan sent by the surgical robot; and verifying whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction; If yes, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed; If not, continue to identify the user's operation on the virtual console until the surgical plan is completed; And it also includes: after sending a control instruction to the surgical robot, receiving the verification result of the control instruction by the surgical robot; when the verification result is unsafe, displaying prompt information to the user through the mixed reality device, and the prompt information is used to prompt that the control instruction is not executed.

6. The surgical robot according to claim 5, characterized in that: The surgical robot is also used to determine a preset surgical plan and send the surgical plan to the mixed reality device.

7. A control device based on mixed reality, characterized in that: include: A rendering module, used for rendering a virtual console and projecting it onto a mixed reality device worn by a user performing the surgery, so as to display it to the user; A monitoring module, configured to determine, based on the monitored gesture operation of the user on the virtual console, an identifier of a controller in the virtual console operated by the user; A control module, used for sending a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction; A display module, used for receiving the operation result of the surgical robot according to the control instruction and displaying it to the user; A receiving module, used for receiving the surgical plan sent by the surgical robot; Verifying whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction; If yes, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed; if no, continue to identify the user's operation on the virtual console until the surgical plan is completed; A prompt module is used to receive the verification result of the control instruction by the surgical robot after sending the control instruction to the surgical robot; when the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device, and the prompt message is used to prompt that the control instruction has not been executed.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 2 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 2 is implemented.

Images