Control method based on mixed reality, mixed reality equipment and surgical robot

Abstract

The invention discloses a control method based on mixed reality, mixed reality equipment and a surgical robot. A virtual console is rendered and projected on mixed reality equipment worn by a user executing an operation so as to be displayed to the user, the operation of the user on the virtual console is monitored, a controller identifier in the virtual console operated by the user is determined, and a control instruction is sent to a surgical robot according to the controller identifier and the operation. And enabling the surgical robot to operate according to the control instruction to execute surgery. In the operation process, whether the operation of the user on the virtual console is recognized or not can be judged according to the preset operation plan and the operation result returned by the operation robot, so that the situation that the user triggers the operation by mistake can be avoided, and then the safety of the operation process is guaranteed.

Description

Technical Field

[0001] This specification relates to the field of medical technology, and in particular to a mixed reality-based control method, mixed reality equipment, and a surgical robot. Background Art

[0002] At present, surgical robots have been widely used in the medical field. In surgical scenarios, since surgical robots perform surgical operations (such as suturing, nailing, etc.) more stably than manual operations and can reduce the workload of doctors, the application of surgical robots in surgery has become a hot topic.

[0003] However, to meet surgical sterility requirements, doctors can remotely control surgical robots through a contactless virtual console, completely eliminating the sterile membrane's limitations on operational flexibility. However, the virtual console's continuous monitoring of gestures can generate false trigger signals due to environmental interference or unconscious user movements, causing the surgical robot to perform operations that deviate from the planned procedure. Especially in complex surgical scenarios, failure to promptly interrupt non-essential command input can put patients at risk of unplanned procedures. Summary of the Invention

[0004] This specification provides a mixed reality-based control method, mixed reality device, and surgical robot to partially solve the above-mentioned problems existing in the prior art.

[0005] This manual adopts the following technical solutions:

[0006] This specification provides a manipulation method based on mixed reality, which is applied to a mixed reality device capable of communicating with a surgical robot. The method includes:

[0007] Rendering and displaying a virtual console, wherein at least one controller is displayed on the virtual console;

[0008] determining a controller identifier of a controller operated by the user according to a monitored user operation on the virtual console;

[0009] Sending a control instruction to the surgical robot so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs, and returns an operation result; the control instruction is determined according to the controller identifier and the operation;

[0010] The operation result is received, and when it is determined based on the operation result and a pre-acquired operation plan that the operation plan is not completed, the user's operation on the virtual console is continuously identified until the operation plan is completed.

[0011] Optionally, the step of rendering the virtual console specifically includes:

[0012] receiving a surgical plan sent by the surgical robot;

[0013] determining the surgery to be performed according to the surgical plan;

[0014] According to the operation to be performed, selecting at least some relevant parameters from preset relevant parameters;

[0015] According to the selected relevant parameters, a virtual console is rendered, and the function of each controller in the virtual console is determined.

[0016] Optionally, the control instruction carries at least an operation parameter, and the operation parameter includes at least one of an operation position, a current position, a starting operation position, a target operation position, an operation speed, and an operation action;

[0017] Sending a control instruction to the surgical robot so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs, and returning an operation result, specifically including:

[0018] A control instruction is sent to the surgical robot through the communication connection so that the surgical robot performs the target operation indicated by the control instruction according to the operation parameters and returns the operation result; the target operation is the operation corresponding to the function of the controller to which the controller identifier triggered by the operation belongs.

[0019] Optionally, the operation parameter is determined based on at least one of the identified operation mode, operation force, operation frequency, operation amplitude, and operation speed of the operation.

[0020] Optionally, sending a control instruction to the surgical robot through the communication connection so that the surgical robot performs a target operation indicated by the control instruction according to the operation parameters and returns an operation result, specifically including:

[0021] A control instruction is sent to the surgical robot through the communication connection so that the surgical robot moves to the operating position and performs the target operation indicated by the control instruction after verifying that the current position is consistent with the current actual position of the surgical robot, and returns the operation result.

[0022] Optionally, sending a control instruction to the surgical robot through the communication connection so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs and returns an operation result, specifically including:

[0023] A control instruction is sent to the surgical robot through the communication connection, so that the surgical robot performs the target operation and returns the operation result when it determines that the target operation indicated by the control instruction is within the range allowed by the preset surgical plan; the target operation is the operation corresponding to the function of the controller to which the controller identifier triggered by the operation belongs.

[0024] Optionally, the method further includes:

[0025] receiving a status parameter sent by the surgical robot via the communication connection;

[0026] When it is determined according to the state parameter that the device of the actuator on the surgical robot body is switched, the device type of the device of the switched actuator is determined;

[0027] The operations triggered by each controller in the virtual console are updated according to the device type and the pre-configured correspondence; the correspondence is the correspondence between the surgical robot that can be switched during surgery and the functions of the virtual console.

[0028] Optionally, the method further includes:

[0029] receiving an execution status of a target operation indicated by the control instruction sent by the surgical robot;

[0030] The user is prompted according to the execution status.

[0031] This specification provides a mixed reality device, which includes: a communication module and a processor, wherein the communication module is used to communicate with a surgical robot; and the processor is used to execute the method described in the above-mentioned mixed reality-based manipulation method.

[0032] This specification provides a surgical robot, which includes: a communication module and a processor, wherein the communication module is used to communicate with a mixed reality device; and the processor is used to execute the method described in the above-mentioned mixed reality-based manipulation method.

[0033] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0034] In the mixed reality-based control method provided in this specification, a virtual console is rendered and displayed, wherein at least one controller is displayed on the virtual console. Based on the monitored user operation on the virtual console, the controller identifier of the controller operated by the user is determined, and a control instruction is sent to the surgical robot to enable the surgical robot to perform the operation corresponding to the function of the controller to which the controller identifier belongs, and return the operation result. The control instruction here is determined based on the controller identifier and the user's operation on the virtual console. The operation result is received, and when it is determined that the surgical plan is not completed based on the operation result and the pre-acquired surgical plan, the user's operation on the virtual console continues to be identified until the surgical plan is completed.

[0035] It can be seen from the above method that this method can control the surgical robot by manipulating the virtual console rendered by the mixed reality device without contact with the mixed reality device as a control device. During the operation, the method can determine whether to recognize the user's operation on the virtual console based on the preset surgical plan and the operation results returned by the surgical robot, thereby avoiding the user's accidental triggering of the operation and ensuring the safety of the surgical process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0037] Figure 1 A schematic diagram of a mixed reality-based control method in this manual;

[0038] Figure 2 A schematic diagram of a mixed reality-based surgical scene provided in this manual;

[0039] Figure 3 A schematic diagram of the display content of a mixed reality device provided in this manual;

[0040] Figure 4 A schematic diagram of the interaction process between a mixed reality device and a surgical robot provided in this manual;

[0041] Figure 5 A schematic diagram of a mixed reality-based control device provided in this manual;

[0042] Figure 6 This is a structural diagram of an electronic device provided in this manual. DETAILED DESCRIPTION

[0043] In traditional surgical scenarios, since the operation needs to be performed in a sterile environment, in order to reduce the patient's risk of infection, when the doctor operates the console in the operating room to control the surgical robot, the console and the surgical robot need to be sterilized, for example, by disinfecting the surgical robot and the console with disinfectants, or covering them with sterile film.

[0044] Surgical robots are devices that perform surgical procedures on the surface of a patient's skin or inside their body. They are typically easier to clean and disinfect than control consoles. Control consoles, however, need to integrate multiple functions, and the sizes and complexities of control consoles adapted for different surgical scenarios vary. While the surfaces of some control consoles are relatively easy to clean with disinfectants, internal areas such as crevices and inner walls are difficult to completely sterilize. Furthermore, patient fluids and medications released during surgery can easily contaminate the console, making subsequent cleaning more difficult and requiring increased cleaning effort. Cleaning agents can also corrode the console, and with repeated cleanings, the corrosion can become more severe, potentially causing partial malfunction of the console's controllers, increasing maintenance costs. Therefore, a sterile treatment is typically performed by covering the console with a sterile film.

[0045] However, the sensitivity of a sterile-film-covered console can be reduced, especially when the console includes movable controls such as joysticks. The sterile film can restrict the movement of the controls, limiting the surgeon's control of the surgical robot. Furthermore, if the sterile film is damaged while the surgeon is manually operating the sterile-film-covered console, the console can be exposed, rendering the sterile film ineffective, increasing the risk of infection and exposing the patient to a dangerous environment.

[0046] In another surgical scenario, the surgeon and the control console are both located away from the operating room. The surgeon remotely controls the surgical robot through the control console to perform operations on the patient's surgical site, and observes the patient or the surgical robot's actions through a transparent medium or surveillance camera connected to the operating room. However, in this method, since the surgeon is not physically present in the operating room, their line of sight and field of view are limited by factors such as obstruction and angle, making it impossible to observe the patient and the execution of the surgery from an optimal perspective. This can even lead to visual illusions due to angle issues, resulting in errors in the operation of the surgical robot, increasing the risk of surgery and even causing surgical failure.

[0047] To at least partially address these issues, the embodiments provided herein are designed to place the surgeon and the control console within the operating room, allowing the surgeon to observe the patient and the robotic manipulation of the surgical site from any perspective. Furthermore, the control console referred to herein is a virtual console rendered by a mixed reality device, specifically a head-mounted mixed reality device, such as smart glasses integrated with mixed reality technology.

[0048] Mixed Reality (MR) devices are capable of rendering and displaying a virtual 3D scene based on the wearer's physical environment, enabling interaction between the real and virtual worlds. For example, a MR device can display a blend of real and virtual environments to the wearer, recognize gestures, sounds, and other information from the wearer, and respond accordingly. For example, a recognized rotation gesture can cause a virtual 3D object displayed to the wearer to rotate in response to the wearer's gesture.

[0049] In the embodiments provided in this specification, users (such as doctors and other medical staff) can wear a mixed reality device connected to a surgical robot during surgery. And control the surgical robot by operating the virtual console rendered by the mixed reality device. The mixed reality device can be covered with a sterile film, and the user only wears the mixed reality device as a control device. After wearing it, there is no need to operate or touch the mixed reality device during surgery. Instead, the user controls the surgical robot by manipulating the virtual console. The console contains several controllers for controlling the surgical robot to perform corresponding operations and functions. The user controls the surgical robot by manipulating the controller in the virtual console. Since the virtual console is virtual, it does not need to be sterilized, and the operation of the controller will not be restricted due to the covering of the console by the sterile film, that is, there will be no inflexible operation of the controller, and the sterile film will not be damaged due to manual operation of the console, increasing the risk of infection for the patient.

[0050] It should be noted that in this specification, the surgical robot and the mixed reality device can communicate through a wired connection or a wireless connection, which can be set according to needs and is not limited in this specification.

[0051] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0052] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0053] Figure 1 This is a schematic diagram of a mixed reality-based control method in this specification, which specifically includes the following steps:

[0054] S100: Rendering and displaying a virtual console, where at least one controller is displayed on the virtual console.

[0055] In this specification, the mixed reality-based manipulation method can be performed by a mixed reality device worn by a user performing surgery. The mixed reality device is used to control the surgical robot. The surgical robot is used to perform surgery on a patient under the control of the user via the mixed reality device.

[0056] In one or more embodiments of this specification, the mixed reality device may first render a virtual control console and project the virtual control console onto the mixed reality device worn by the user performing the surgery for display to the user. Therefore, the mixed reality device may include at least a display device. If the mixed reality device is a pair of smart glasses, the display device may be a lens. The user can manipulate the controllers within the virtual control console viewed through the mixed reality device to control the surgical robot.

[0057] The virtual console is a console rendered by the mixed reality device that includes several virtual controllers, each of which is used to implement at least one function or operation of the surgical robot. For example, the controllers of the virtual console may include: a controller for controlling the up and down movement of the surgical robot, a controller for controlling the surgical robot to perform cutting operations corresponding to the surgery, a controller for controlling the surgical robot to perform nail placement operations, suturing operations, puncture operations, and other operations corresponding to the surgery, and so on. Of course, the controllers included in the console mentioned in this specification are only examples and not limitations. They can be flexibly set according to the nature of the surgery and the needs of the surgery, and this specification does not impose any restrictions on this.

[0058] In addition, the function, form and number of controllers included in the virtual console can be set as needed, and this manual does not impose any restrictions here. The relevant parameters for rendering the virtual console can be pre-designed and stored in the mixed reality device according to the nature of the surgery to be performed and the surgical needs. When the mixed reality device is started, the mixed reality device can render the virtual console for execution. That is, according to the surgery to be performed, the virtual console can be pre-set to a fixed form, and according to the surgery to be performed, the corresponding functions of each controller in the virtual console are assigned. For example, assuming that the surgery to be performed is a surgery to place nails in multiple bone positions of the patient, the surgical robot controlled by the mixed reality device needs to have the ability to move and place nails. Accordingly, the virtual console may include a controller for controlling the movement of the surgical robot and a controller for controlling the surgical robot to perform the nailing operation.

[0059] S102: Determine a controller identifier of a controller operated by the user according to the monitored user operation on the virtual console.

[0060] In this specification, each controller in the virtual console has a unique controller identifier. The mixed reality device can monitor the user's gestures on the virtual console in real time and determine the controller identifier in the virtual console operated by the user, so as to determine the control instructions issued to the surgical robot in subsequent steps, that is, determine the operation to be performed by the surgical robot.

[0061] In one or more embodiments of this specification, the user may also control the mixed reality device through voice to send control instructions to the surgical robot. The mixed reality device may also perform voice recognition on the voice data collected by the user to determine the operating parameters contained in the voice data.

[0062] Among them, when performing voice recognition, existing voice recognition technology can be used, and this manual will not elaborate on it here.

[0063] The mixed reality device can also use voice prompts to inform the user of the results of the surgical robot's execution of the control command, the status parameters of the surgical robot, the results of the surgical robot's verification of the control command, etc. Of course, other content related to the surgical process can also be included, and this manual does not limit this.

[0064] In one or more embodiments of the present specification, the mixed reality device may also monitor user-triggered operations for controlling the surgical robot in other ways, for example, by tracking the user's eye movements through sensors, etc.

[0065] S104: Sending a control instruction to the surgical robot so that the surgical robot executes an operation corresponding to the function of the controller to which the controller identifier belongs, and returns an operation result; the control instruction is determined according to the controller identifier and the operation.

[0066] S106: Receive the operation result, and if it is determined that the surgical plan is not completed based on the operation result and the pre-acquired surgical plan, continue to identify the user's operation on the virtual console until the surgical plan is completed.

[0067] In one or more embodiments of the present specification, after determining the controller identifier of the user operation and the user's gesture operation, the mixed reality device can send a control instruction to the surgical robot based on the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction and performs the target operation indicated by the control instruction.

[0068] The control instruction carries at least an operation parameter, which is used to control the surgical robot to perform a target operation indicated by the control instruction. The target operation is an operation corresponding to the function of the controller to which the controller identifier belongs, triggered by a user's gesture operation.

[0069] In one or more embodiments of the present specification, the surgical robot may operate according to the control instruction received from the mixed reality device, i.e., perform the target operation indicated by the control instruction, and send the operation result, i.e., the execution result of the target operation, to the mixed reality device.

[0070] Therefore, in one or more embodiments of the present specification, the mixed reality device may also receive the operating result of the surgical robot according to the control instruction and display the operating result to the user, so that the user can perform corresponding operations based on the operating result displayed by the mixed reality device.

[0071] Alternatively, after receiving the operation result, the mixed reality device may also convey the operation result to the user through voice notification.

[0072] The operation results include success and failure. If the operation result is success, the user can continue with subsequent operations or end the operation. If the operation result is failure, the user can regain control of the controller in the virtual console by initiating gestures or perform other operations, such as terminating the operation, checking the surgical robot, or performing other operations on the patient.

[0073] based on Figure 1 The mixed reality-based control method shown renders a virtual console and projects it onto a mixed reality device worn by the user performing the operation to display it to the user, monitors the user's gesture operations on the virtual console, determines the controller identifier in the virtual console operated by the user, and sends control instructions to the surgical robot based on the controller identifier and gesture operations, so that the surgical robot operates according to the control instructions to perform the operation.

[0074] It can be seen from the above method that this method can control the surgical robot by manipulating the virtual console rendered by the mixed reality device without contact with the mixed reality device as a control device. The control of the virtual console will not be restricted by the sterile membrane and will not be inflexible, and there will be no situation where the sterile membrane is damaged due to manual operation, thereby ensuring the safety of the surgical process.

[0075] In addition, in one or more embodiments of this specification, because the mixed reality device needs to detect user gestures, the mixed reality device may be equipped with a sensor. The sensor can be configured as needed, for example, it can be a visual sensor, an infrared sensor, etc. Of course, other sensors are also possible, as long as they can recognize user gestures on the controller. This specification does not limit this.

[0076] Based on this, before determining the controller identifier in the virtual console operated by the user based on the monitored user gesture operation on the virtual console, the mixed reality device may also monitor the user's hand pose for several frames through sensors. Based on the monitored hand poses for several frames and the position of the virtual console, the user's gesture operation on the virtual console is determined.

[0077] In addition, in one or more embodiments of the present specification, the surgical robot may also store the surgical plan of the operation to be performed, and send the surgical plan to the mixed reality device. The mixed reality device can also receive the surgical plan sent by the surgical robot, and after receiving the results of the surgical robot's operation of the control instruction and the surgical plan, verify whether the surgical plan is completed based on the received results of the surgical robot's operation according to the control instruction. If so, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed. Avoid performing operations outside the surgical plan or other risky operations on the patient due to monitoring of gesture operations triggered by the user in error, thereby further ensuring the safety of the operation.

[0078] In one or more embodiments of the present specification, if the surgical plan is not completed, the mixed reality device may continue to recognize the user's operation on the virtual console until the surgical plan is completed.

[0079] Among them, the surgical plan may include pre-planned specific operations to be performed on the patient when performing the operation to be performed, the tools, quantity, operation location, etc., or it may also include relevant data of the patient corresponding to the operation, such as electronic computed tomography (CT) images, etc. Taking suturing surgery as an example, the surgical plan of the operation may include suturing, suturing location, suturing method, etc. The surgical plan may include a whitelist and / or blacklist of operations to be performed on the patient. The surgical robot may only execute control instructions corresponding to the operations in the whitelist sent by the mixed reality device, or execute control instructions corresponding to other operations other than the prohibited operations in the blacklist. The specific settings can be made as needed, and this manual does not limit it here.

[0080] In one or more embodiments of the present specification, after receiving the surgical plan, the mixed reality device can display the surgical plan to the user, so that the user can perform the surgery corresponding to the surgical plan on the patient according to the surgical plan, or refer to the surgical plan and perform the surgery in combination with the patient's status during the operation.

[0081] In this specification, the surgical robot can also send the current state parameters of the surgical robot to the mixed reality device in real time when it is started or during the execution of the operation. The surgical robot can be configured with a chip encapsulated with a program for realizing the functions of the surgical robot, and the state parameters include at least: the running state of the program and the device state and device parameters of the surgical robot. For example, the running state may include: operating failure, inability to perform surgery, executable surgery, etc. The device state may include whether the surgical robot is in an executable state (that is, whether it can respond to the control instructions of the mixed reality device and perform the target operation), whether it is faulty, etc. The device parameters include at least: the current posture of the surgical robot, the movable range of the surgical robot, the moving speed range of the surgical robot, etc.

[0082] Since it may be necessary to switch different surgical robots or components configured on the surgical robot during surgery to achieve different functions in order to perform different surgical operations on the patient. For example, the surgical robot may be composed of a body and an actuator, and the surgical robot is a device that can switch different actuators, that is, a device that can switch and dock different actuators on the body. Different actuators correspond to different functions, so that when the surgical robot docks different actuators, the meaning of the instructions for controlling the surgical robot that can be triggered by the virtual console of the mixed reality device is different. Therefore, the device parameters of the surgical robot may also include the device type (such as a scalpel, suture needle, puncture needle, C-arm, etc.). Accordingly, the mixed reality device can be configured with a correspondence between the surgical robot that can be switched during surgery and the function of the virtual console, so that when the surgical robot or actuator is switched, the mixed reality device can further update the meaning of the operation that can be triggered by the controller in the virtual console according to the correspondence between the switched surgical robot and the function of the controller in the virtual console, so that the mixed reality device can determine the accurate control instructions based on the recognized user gesture operation and controller identification.

[0083] In one or more embodiments of the present specification, the mixed reality device may also display the current status parameters of the surgical robot to the user or prompt the user through voice when receiving the current status parameters of the surgical robot sent by the surgical robot.

[0084] In one or more embodiments of the present specification, the surgical robot may send the surgical plan to the mixed reality device upon startup. In step S100, the virtual console rendered by the mixed reality device may be a virtual console adapted for the surgery to be performed, rendered based on the received surgical plan. Alternatively, the mixed reality device may store configuration parameters for virtual consoles adapted for a variety of surgeries. Upon receiving the surgical plan, the mixed reality device may also determine a virtual console adapted for the surgical plan from a number of pre-configured virtual consoles and render the virtual console.

[0085] Furthermore, since surgical safety is a key factor in the execution of a procedure, in one or more embodiments of this specification, the surgical robot may verify control instructions received from the mixed reality device based at least on the surgical plan to determine whether the control instructions are safe. The robot then sends the verification result to the mixed reality device. If the verification result indicates unsafe, the surgical robot does not perform the target operation specified by the control instruction. If the verification result indicates safe, the surgical robot performs the target operation specified by the control instruction.

[0086] In one or more embodiments of the present specification, when the surgical robot verifies whether the target operation indicated by the control instruction is safe according to the surgical plan, it can determine whether the target operation is within the range allowed by the surgical plan. If not, it is determined that the target operation indicated by the control instruction is unsafe; if so, it is determined that the target operation indicated by the control instruction is safe.

[0087] In one or more embodiments of the present specification, when the surgical robot verifies whether the target operation indicated by the control instruction is safe according to the surgical plan, it can determine whether the target operation is within the range prohibited by the surgical plan. If so, it is determined that the target operation indicated by the control instruction is unsafe; if not, it is determined that the target operation indicated by the control instruction is safe.

[0088] Specifically, taking the example of a nail placement operation where the operation to be performed is a nail placement operation, the surgical plan may include the target nail placement positions in the patient's surgical site where the nail placement operation can be performed and the total number of nails to be placed in the operation to be performed. Assuming that the target operation is to place a nail at position A, the surgical robot may verify whether position A is a target nail placement position in a safe position according to the surgical plan upon receiving the control instruction, and verify whether the number of nails placed on the patient as of now is not greater than the total number of nails in the surgical plan. If the verification results are all yes, it is determined that the target operation indicated by the control instruction is safe; otherwise, it is determined that the target operation indicated by the control instruction is unsafe.

[0089] Since the surgical robot may need to perform operations on multiple positions during the operation, the surgical robot has the ability to move, and the movement can be overall movement or local movement. For example, the surgical robot can include a fixed base and a movable robotic arm.

[0090] Accordingly, when the surgical robot needs to be controlled, the user can manipulate the virtual control console used to control the movement of the surgical robot, and the mixed reality device can send control instructions to the surgical robot. The operation parameters carried in the control instructions include at least the operation position for performing the target operation, that is, the target position to which the surgical robot needs to be controlled to move.

[0091] The surgical robot can move to the operating position according to the control instruction.

[0092] In addition, the surgical robot can also determine whether it has moved to the operation position when executing the target operation indicated by the control instruction. If so, it executes the target operation indicated by the control instruction; if not, it continues to move to the operation position.

[0093] Alternatively, the operating parameter may include the current position of the surgical robot and the operating position for performing the target operation. In one or more embodiments of the present specification, in order to ensure that the mixed reality device and the surgical robot maintain a good synchronization relationship, the surgical robot is also used to verify whether the current position is consistent with the current actual position of the surgical robot when receiving the control instruction. If so, it moves to the operating position; if not, it sends an error message to the mixed reality device. This avoids the situation where the mixed reality device and the surgical robot fail to maintain good synchronization due to unnoticed faults or errors during the execution of the operation, and the surgical robot has a wrong understanding of its current position, which leads to risks in the operation.

[0094] In one or more embodiments of the present specification, the mixed reality device may also receive error information sent by the surgical robot and prompt the user based on the error information.

[0095] Alternatively, during a surgical procedure, the operation to be performed on a patient may not be a single operation, but rather a sequence of operations, such as suturing the skin from position A to position B. In this case, the target operation indicated by the control instruction involves at least two target positions, i.e., at least two operating positions. Therefore, the operating parameters may also include at least the surgical robot's starting operating position and target operating position, or the surgical robot's current position, starting operating position, and target operating position.

[0096] In one or more embodiments of the present specification, the operating parameter may further include an operating speed. The surgical robot is further configured to move to the operating position, or move from the current position to the starting operating position, and then from the starting operating position to the target operating position, at the operating speed when performing the target operation indicated by the control instruction.

[0097] In one or more embodiments of the present specification, the operation parameter may further include an operation action and an operation speed. The surgical robot may further perform the target operation at the operation speed when performing the target operation indicated by the control instruction.

[0098] To sum up, the operation parameters may include: operation position, current position, starting operation position, target operation position, operation speed and a combination of one or more operation actions. Of course, they may also include others, such as operation force, operation amplitude, etc., which are not limited in this manual.

[0099] In one or more embodiments of this specification, the mixed reality device may also determine operating parameters for the surgical robot based on a combination of one or more of the user's recognized manipulation of the virtual console, manipulation force, manipulation frequency, manipulation amplitude, and manipulation speed. Of course, other methods may also be used, and this specification does not limit this.

[0100] Since it is difficult for the user to observe with the naked eye whether some operations of the surgical robot on the patient (such as subcutaneous operations) are completed or successfully executed, in one or more embodiments of the present specification, the surgical robot can also feedback the execution status of the target operation to the mixed reality device when executing the target operation indicated by the control instruction, and the mixed reality device can receive the execution status sent by the surgical robot and prompt the user. The execution status includes: in progress and completed. Alternatively, the execution status may correspond to the target operation. For example, if the target operation is to move from position A to position B, then in the process of the surgical robot moving from position A to position B, before reaching position B, the execution status may be "moving". The execution status of the completed execution is used to prompt the user that the surgical robot no longer continues to perform the target operation, and does not necessarily mean that the target operation is successfully executed.

[0101] In one or more embodiments of this specification, the mixed reality device may also receive the surgical robot's verification result of the control instruction. If the verification result is unsafe, the mixed reality device may display a prompt message to the user, indicating that the control instruction was not executed.

[0102] It should be noted that in one or more embodiments of this specification, the target operation can be a real-time operation or a process operation that needs to be performed over a time interval. When the target operation is a real-time operation, the user can continuously control the controller through a number of gesture operations, and observe the surgical robot continuously performing a number of target operations corresponding to the gesture operations through the mixed reality device. When the target operation is a process operation that needs to be performed over a time interval, the user can intermittently control the controller through gesture operations, and initiate a new gesture operation after observing the surgical robot perform the target operation corresponding to the gesture operation through the mixed reality device.

[0103] In this specification, the structure and composition of the mixed reality device and the surgical robot are not limited. For example, the surgical robot may include an actuator and a drive module. The drive module can be used to control the actuator to perform the target operation indicated by the control instruction based on the operating parameters carried by the received control instruction. Because the surgical robot needs to communicate with the mixed reality device, the surgical robot may also include a communication module. Accordingly, the mixed reality device may also include a communication module for communicating with the surgical robot.

[0104] Figure 2 This specification provides a schematic diagram of a mixed reality-based surgical scene. As shown in the figure, the system includes a mixed reality device D and a surgical robot K. The surgical robot includes an actuator B, a drive module B01, a communication module C, and a chip C01 encapsulated with a program that implements the functions of the surgical robot. A represents the patient, G represents the surgical site, and H represents the bed, which is located in the sterile area. It can be seen that the mixed reality device D is capable of rendering a mixed control environment that includes at least a virtual console E. The user F wearing the mixed reality device D can control the surgical robot K by controlling the virtual console E. ①, ②, and ③ respectively represent several positions of the actuator B during its movement.

[0105] In one or more embodiments of this specification, the content displayed to the user by the mixed reality device includes at least a virtual console. Of course, in addition to the virtual console, other content may also be displayed, such as Figure 3 As shown, of course, the specific content to be displayed can be set as needed, and this manual does not limit it here.

[0106] Figure 3This is a schematic diagram of the display content of a mixed reality device provided in this specification. As shown in the figure, the mixed reality device shows the field of view presented to the user. It can be seen that in addition to the real environment background seen by the user, the field of view also includes a CT image of the patient taken before surgery and a three-dimensional model of the patient's bones constructed based on the CT image. The user can refer to the displayed content and control the surgical robot to perform the operation through the mixed reality device. The dotted box is a virtual console rendered by the mixed reality device, which contains several controllers.

[0107] This manual also provides the interaction process between the mixed reality device and the surgical robot. Figure 4 .

[0108] Figure 4 This is a schematic diagram of the interaction process between a mixed reality device and a surgical robot provided in this manual. The interaction process includes:

[0109] S300: The mixed reality device renders a virtual console and displays the virtual console to a user wearing the mixed reality device.

[0110] S301: The surgical robot determines the preoperative plan and its own equipment parameters.

[0111] S302: The surgical robot sends the determined preoperative plan and device parameters to the mixed reality device.

[0112] S303: The mixed reality device monitors the user's gesture operation and determines the controller identifier of the user's operation.

[0113] S304: The mixed reality device sends a control instruction carrying operation parameters to the surgical robot based on the gesture operation and the controller identifier. The control instruction is used to instruct the surgical robot to perform the target operation indicated by the control instruction according to the operation parameters.

[0114] S305: The surgical robot verifies whether the received control instruction is safe at least according to the surgical plan. If so, steps S307 to S308 and S310 are executed; otherwise, S306 is executed.

[0115] S306: The surgical robot feeds back the verification result to the mixed reality device.

[0116] S307: The surgical robot feeds back the verification result to the mixed reality device.

[0117] S308: The surgical robot operates according to the operating parameters carried by the control instruction and performs the target operation indicated by the control instruction.

[0118] S309: The mixed reality device displays the received verification result to the user.

[0119] S310: The surgical robot feeds back the operating results to the mixed reality device.

[0120] S311: The mixed reality device displays the operation result to the user.

[0121] S312: The mixed reality device verifies whether the surgical plan has been completed based on the operation result. If so, step S313 is executed; if not, step S303 is repeated.

[0122] S313: Prompt the user that the surgical plan is finished, and stop monitoring the user's gesture operation.

[0123] It should be noted that although the mixed reality device and the surgical robot may interact multiple times during the surgical procedure, in steps S300 to S313, if the steps performed by the mixed reality device and the surgical robot do not affect, interfere with, or contradict each other, the two can perform the corresponding steps simultaneously or sequentially, and this specification does not impose any restrictions. The same applies to steps S100 to S108.

[0124] based on Figure 1 In addition to the mixed reality-based control method shown, this specification also provides a mixed reality device.

[0125] The mixed reality device is used to control a surgical robot and is worn by a user performing surgery, wherein:

[0126] The mixed reality device is used to render a virtual console and display it to the user. Based on the user's gestures detected on the virtual console, the device determines the controller identifier in the virtual console operated by the user. Based on the controller identifier and the gestures, the device sends control instructions to the surgical robot, receives the results of the surgical robot's operation based on the control instructions, and displays the results to the user.

[0127] In one or more embodiments of the present specification, the mixed reality device may be provided with a sensor. The mixed reality device is further configured to monitor the user's hand gestures in a plurality of frames via the sensor, and determine the user's gesture operation on the virtual console based on the monitored hand gestures in the plurality of frames and the position of the virtual console.

[0128] The mixed reality device also receives the surgical plan sent by the surgical robot and verifies whether the surgical plan is complete based on the robot's execution of the control instructions. When the surgical plan is complete, the device stops monitoring the user's gestures on the virtual console and notifies the user that the surgical plan is complete.

[0129] In one or more embodiments of the present specification, the mixed reality device is further configured to continue recognizing the user's operations on the virtual console when the surgical plan is not completed until the surgical plan is completed.

[0130] In one or more embodiments of the present specification, the mixed reality device is further configured to receive a verification result of a control instruction by the surgical robot. If the verification result is unsafe, the device displays a prompt message to the user, re-monitors the user's gesture operation on the virtual console, and sends a control instruction to the surgical robot based on the re-identified user gesture operation on the virtual console and the controller identifier in the virtual console operated by the user. The prompt message indicates that the control instruction was not executed.

[0131] This specification also provides a surgical robot that is used to receive control instructions sent by a mixed reality device worn by a user performing surgery, operate according to the control instructions, and send the operation results to the mixed reality device. The control instructions are determined by the mixed reality device based on the user's gesture operation on the rendered virtual console and the controller identifier in the virtual console operated by the user.

[0132] In this specification, the surgical robot is also used to determine a preset surgical plan and send the surgical plan to a mixed reality device.

[0133] In one or more embodiments of the present specification, the surgical robot is further configured to verify the received control instructions based at least on the surgical plan, and to send the verification results of the control instructions to the mixed reality device.

[0134] It should be noted that the corresponding contents of the mixed reality-based control method, mixed reality device and surgical robot provided in the above-mentioned specification can be referenced to each other.

[0135] This manual also provides corresponding mixed reality-based control devices, such as Figure 5 shown.

[0136] Figure 5 This is a schematic diagram of a mixed reality-based control device provided in this manual, which includes:

[0137] a rendering module 200 , configured to render a virtual console and project it onto a mixed reality device worn by a user performing surgery, so as to display it to the user;

[0138] A monitoring module 201 is configured to determine an identifier of a controller in the virtual console operated by the user based on a monitored gesture operation of the user on the virtual console;

[0139] A control module 202 is configured to send a control instruction to the surgical robot according to the controller identifier and the gesture operation, so that the surgical robot operates according to the control instruction;

[0140] The display module 203 is used to receive the operating results of the surgical robot according to the control instructions and display them to the user.

[0141] Optionally, a sensor is provided on the mixed reality device, and the monitoring module 201 is further used to monitor the hand postures of the user in several frames through the sensor, and determine the user's gesture operation on the virtual console based on the monitored hand postures of the several frames and the posture of the virtual console.

[0142] The device further comprises:

[0143] The receiving module 204 is used to receive the surgical plan sent by the surgical robot, and verify whether the surgical plan is completed based on the received operation result of the surgical robot according to the control instruction. If so, stop monitoring the user's gesture operation on the virtual console, and prompt the user through the mixed reality device that the surgical plan is completed. If not, continue to identify the user's operation on the virtual console until the surgical plan is completed.

[0144] The device further comprises:

[0145] The prompt module 205 is used to receive the verification result of the surgical robot on the control instruction. When the verification result is unsafe, a prompt message is displayed to the user through the mixed reality device, where the prompt message is used to prompt that the control instruction has not been executed.

[0146] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provided is a control method based on mixed reality.

[0147] This manual also provides Figure 6 The structural diagram of the electronic device shown in FIG. Figure 6 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provided is a control method based on mixed reality.

[0148] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0149] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0150] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also understand that in addition to implementing the controller in pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0151] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0152] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0153] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0158] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0159] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0160] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0161] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0163] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0164] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A control method based on mixed reality, characterized in that: The method is applied to a mixed reality device capable of communicating with a surgical robot, and includes: Rendering and displaying a virtual console, wherein at least one controller is displayed on the virtual console; determining a controller identifier of a controller operated by the user according to a monitored user operation on the virtual console; Sending a control instruction to the surgical robot so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs, and returns an operation result; the control instruction is determined according to the controller identifier and the operation; The operation result is received, and when it is determined based on the operation result and a pre-acquired operation plan that the operation plan is not completed, the user's operation on the virtual console is continuously identified until the operation plan is completed.

2. The method according to claim 1, wherein The steps for rendering a virtual console include: receiving a surgical plan sent by the surgical robot; determining the surgery to be performed according to the surgical plan; According to the operation to be performed, selecting at least some relevant parameters from preset relevant parameters; According to the selected relevant parameters, a virtual console is rendered, and the function of each controller in the virtual console is determined.

3. The method according to claim 1, wherein The control instruction carries at least one operation parameter, and the operation parameter includes at least one of an operation position, a current position, a starting operation position, a target operation position, an operation speed, and an operation action; Sending a control instruction to the surgical robot so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs, and returning an operation result, specifically including: A control instruction is sent to the surgical robot through the communication connection so that the surgical robot performs the target operation indicated by the control instruction according to the operation parameters and returns the operation result; the target operation is the operation corresponding to the function of the controller to which the controller identifier triggered by the operation belongs.

4. The method according to claim 3, wherein The operation parameter is determined according to at least one of the identified operation mode, operation force, operation frequency, operation amplitude, and operation speed of the operation.

5. The method according to claim 3, wherein Sending a control instruction to the surgical robot through the communication connection, so that the surgical robot performs a target operation indicated by the control instruction according to the operation parameters and returns an operation result, specifically including: A control instruction is sent to the surgical robot through the communication connection so that the surgical robot moves to the operating position and performs the target operation indicated by the control instruction after verifying that the current position is consistent with the current actual position of the surgical robot, and returns the operation result.

6. The method according to claim 1, wherein Sending a control instruction to the surgical robot through the communication connection so that the surgical robot performs an operation corresponding to the function of the controller to which the controller identifier belongs, and returning an operation result, specifically including: A control instruction is sent to the surgical robot through the communication connection, so that the surgical robot performs the target operation and returns the operation result when it determines that the target operation indicated by the control instruction is within the range allowed by the preset surgical plan; the target operation is the operation corresponding to the function of the controller to which the controller identifier triggered by the operation belongs.

7. The method according to claim 1, wherein The method further comprises: receiving a status parameter sent by the surgical robot via the communication connection; When it is determined according to the state parameter that the device of the actuator on the surgical robot body is switched, the device type of the device of the switched actuator is determined; The operations triggered by each controller in the virtual console are updated according to the device type and the pre-configured correspondence; the correspondence is the correspondence between the surgical robot that can be switched during surgery and the functions of the virtual console.

8. The method according to claim 1, wherein The method further comprises: receiving an execution status of a target operation indicated by the control instruction sent by the surgical robot; The user is prompted according to the execution status.

9. A mixed reality device, characterized in that: The mixed reality device includes: a communication module and a processor, the communication module is used to communicate with the surgical robot; the processor is used to execute the method according to any one of claims 1 to 8.

10. A surgical robot, characterized in that: The surgical robot includes: a communication module and a processor, wherein the communication module is used for communication connection with the mixed reality device; and the processor is used for executing the method according to any one of claims 1 to 8.

Images