Surgical robot system and data processing method

Abstract

This specification relates to the field of surgical robot technology, and specifically discloses a surgical robot system and data processing method, wherein the system includes: a mixed reality device, wherein a first mixed reality device captures first image data from the perspective of a operative; a second mixed reality device captures second image data from the perspective of an assistant; a robotic arm, at the end of which a surgical instrument and a third image capture device are mounted; the surgical instrument performs the surgical operation under the control of the robotic arm; the third image capture device captures third image data from the perspective of the robotic arm end; a data processor acquires and integrates and processes the first, second, and third image data; the first mixed reality device acquires and displays the first, second, and third image data; and the second mixed reality device acquires and displays the second and third image data. The above system can provide multiple perspectives, addressing the limitations of the surgical field of view.

Description

Technical Field

[0001] This specification relates to the technical field of surgical robots, and in particular to a surgical robot system and a data processing method. Background Art

[0002] In robotic-assisted surgery, in addition to the surgeon operating the instruments, the assistant is also a crucial participant, and the collaboration between the surgeon and the assistant is crucial to the success of the operation. However, existing surgical instrumentation systems fail to consider the limitations of the surgeon's field of view, the needs of the surgeon's assistant, adaptability to complex environments, and support for remote collaboration. In potential remote operation scenarios, relying solely on the perspective of the robotic arm's end-of-surgery arm is insufficient to address the actual surgical environment and may fail to detect potential interference from the robotic arm's operation and the surrounding environment.

[0003] To address the above issues, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of this specification provide a surgical robot system and a data processing method to solve the problem that the surgical robot operation method in the prior art cannot meet the surgical needs.

[0005] The present invention provides a system for a surgical robot, including:

[0006] The mixed reality device includes a first mixed reality device and a second mixed reality device; the first mixed reality device is worn by a performer and is provided with a first image acquisition device for acquiring first image data from the performer's perspective; the second mixed reality device is worn by an assistant and is provided with a second image acquisition device for acquiring second image data from the assistant's perspective;

[0007] A robotic arm, wherein a surgical instrument and a third image acquisition device are mounted at the end of the robotic arm; the surgical instrument is used to perform a surgical operation under the control of the robotic arm; and the third image acquisition device is used to acquire third image data from the perspective of the end of the robotic arm;

[0008] A data processor is communicatively connected to the robotic arm and the mixed reality device, and is used to acquire and integrate the first image data, the second image data, and the third image data; the first mixed reality device is used to acquire and display the first image data, the second image data, and the third image data processed by the data processor; and the second mixed reality device is used to acquire and display the second image data and the third image data processed by the data processor.

[0009] In one embodiment, the surgical robot system further comprises:

[0010] A force feedback teleoperation device is communicatively connected to the data processor and is used to receive force information during a surgical operation sent by the surgical instrument to the data processor via the robotic arm, so as to provide the user with tactile feedback during the surgical operation based on the force information; the force feedback teleoperation device is also used to capture the user's operating instructions and send the captured operating instructions to the data processor, so that the data processor generates a robotic arm execution instruction based on the operating instruction and sends the robotic arm execution instruction to the robotic arm.

[0011] In one embodiment, the surgical robot system further comprises:

[0012] A scene scanning device is used to scan the environmental information of the operating object and the robotic arm, and send the scanned environmental information to the data processor, so that the data processor establishes a corresponding virtual scene based on the environmental information; the first mixed reality device and the second mixed reality device are also used to receive and display the virtual scene.

[0013] In one embodiment, the data processor is further used to receive operation object planning data; the operation object planning data includes operation object image data and / or surgical operation plan data; the data processor is further used to establish a virtual scene based on the environmental information and the operation object planning data.

[0014] In one embodiment, the scene scanning device includes a natural light camera group and a motion capture camera group;

[0015] The natural light camera group is used to collect environmental image data;

[0016] The motion capture camera group is used to collect motion image data of an object containing a identifiable identifier; the first mixed reality device, the second mixed reality device, the robotic arm and the operating object are all provided with the identifiable identifier.

[0017] In one embodiment, the mixed reality device further includes a third mixed reality device worn by a bystander, and the third mixed reality device is used to receive and display the first image data, the second image data, the third image data and the virtual scene processed by the data processor.

[0018] In one embodiment, the surgical robot system further comprises:

[0019] A surgical evaluation device is used to obtain the first image data, the second image data, the third image data, the virtual scene and the force information from the data processor, and evaluate the surgery corresponding to the surgical operation based on the first image data, the second image data, the third image data, the virtual scene and the force information.

[0020] In one embodiment, the surgical evaluation device evaluates the surgery corresponding to the surgical operation based on a trained machine learning model and outputs a surgical evaluation result.

[0021] The embodiments of this specification also provide a data processing method applied to a surgical robot system, including:

[0022] Acquire first image data, second image data, and third image data; the first image data is image data collected from the perspective of the executor; the second image data is image data collected from the perspective of the assistant; and the third image data is image data collected from the perspective of the end of the robotic arm;

[0023] Integrate the first image data, the second image data, and the third image data to obtain first integrated data; integrate the second image data and the third image data to obtain second integrated data;

[0024] The first integrated data is sent to a first mixed reality device worn by the executor for display; and the second integrated data is sent to a second mixed reality device worn by the assistant for display.

[0025] In one embodiment, the data processing method further includes:

[0026] Obtaining force information of surgical instruments during surgical operations;

[0027] The force information is sent to a force feedback teleoperation device, so that the force feedback teleoperation device provides tactile feedback to the user during the surgical operation based on the force information.

[0028] In one embodiment, the data processing method further includes:

[0029] receiving operation instructions sent by the force feedback teleoperation device;

[0030] A robot arm execution instruction is generated based on the operation instruction, and the robot arm execution instruction is sent to the robot arm.

[0031] In one embodiment, the data processing method further includes:

[0032] Acquiring environmental information of the operating object and the environment in which the robotic arm is located and operating object planning data; the operating object planning data includes operating object image data and / or surgical operation plan data;

[0033] A corresponding virtual scene is established according to the environmental information and the operation object planning data, and the virtual scene is sent to the first mixed reality device and the second mixed reality device for display.

[0034] In one embodiment, the environmental information includes environmental image data and motion image data, the motion image data including position information of an object corresponding to each of a plurality of identifiable identifiers; the plurality of identifiable identifiers include: a robotic arm identifier, an operating object identifier, a first mixed reality device identifier, and a second mixed reality device identifier;

[0035] Accordingly, establishing a corresponding virtual scene according to the environmental information and the operation object planning data includes:

[0036] determining, by calibration, a first transformation relationship between a coordinate system corresponding to the environmental image data and a coordinate system corresponding to the motion image data;

[0037] A second transformation relationship is established between the coordinate system of the robotic arm and the coordinate system of the motion image data by registering the robotic arm; a third transformation relationship is established between the coordinate system of the operating object and the coordinate system of the motion image data by registering the operating object; a fourth transformation relationship is established between the coordinate system of the first mixed reality device and the coordinate system of the motion image data by registering the first mixed reality device; and a fifth transformation relationship is established between the coordinate system of the second mixed reality device and the coordinate system of the motion image data by registering the second mixed reality device.

[0038] A corresponding virtual scene is established based on the environmental image data, the motion image data, the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, the fifth transformation relationship and the operation object planning data.

[0039] The present invention also provides a data processing device for a surgical robot system, including:

[0040] an acquisition module, configured to acquire first image data, second image data, and third image data; the first image data is image data acquired from the perspective of the operator; the second image data is image data acquired from the perspective of the assistant; and the third image data is image data acquired from the perspective of the end of the robotic arm;

[0041] a processing module, configured to integrate the first image data, the second image data, and the third image data to obtain first integrated data; and integrate the second image data and the third image data to obtain second integrated data;

[0042] A sending module is used to send the first integrated data to a first mixed reality device worn by the executor for display; and send the second integrated data to a second mixed reality device worn by the assistant for display.

[0043] An embodiment of this specification also provides a medical device, including a processor and a memory for storing processor-executable instructions, wherein when the processor executes the instructions, the steps of the data processing method applied to the surgical robot system described in any of the above embodiments are implemented.

[0044] An embodiment of this specification also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the steps of the data processing method applied to a surgical robot system described in any of the above embodiments.

[0045] In an embodiment of the present specification, a surgical robot system is provided, including a mixed reality device, a robotic arm and a data processor. The mixed reality device may include a first mixed reality device worn by a performer and a second mixed reality device worn by an assistant. The first mixed reality device is provided with a first image acquisition device, which can acquire first image data from the performer's perspective. The second mixed reality device is provided with a second image acquisition device, which can acquire second image data from the assistant's perspective. A surgical operating instrument and a third image acquisition device are installed at the end of the robotic arm. The surgical operating instrument can perform a surgical operation under the control of the robotic arm. The third image acquisition device can acquire third image data from the perspective of the end of the robotic arm. The data processor is communicatively connected to the robotic arm and the mixed reality device, and can acquire and integrate the first image data, the second image data and the third image data. The first mixed reality device can acquire and display the first image data, the second image data and the third image data processed by the data processor, and the second mixed reality device can acquire and display the second image data and the third image data processed by the data processor. In the above scheme, the assistant's perspective and the perspective of the end of the robotic arm can be introduced into the executor's perspective. The perspective of the end of the robotic arm can display the actual situation of the area to be operated on, and the assistant's perspective can supplement the visible range of the executor's perspective, effectively broadening the executor's field of view, so that the executor can promptly discover possible interference between the robotic arm and the surrounding environment during operation, avoid interference between the robotic arm and the environment, assistant or operation object, and facilitate the executor to control the robotic arm to perform surgery, improve the accuracy of the surgery, and improve the surgical experience of the operation object. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings described herein are used to provide a further understanding of this specification, constitute a part of this specification, and do not constitute a limitation of this specification. In the accompanying drawings:

[0047] Figure 1 A schematic diagram of a surgical robot system according to an embodiment of the present specification is shown;

[0048] Figure 2 A schematic diagram showing the principle of a single scene scanning device in one embodiment of this specification is shown;

[0049] Figure 3 A schematic diagram showing the principle of a scene scanning device group in one embodiment of this specification is shown;

[0050] Figure 4 A schematic diagram of a force feedback teleoperation device according to an embodiment of the present specification is shown;

[0051] Figure 5 A schematic structural diagram of a surgical robot system according to an embodiment of the present specification is shown;

[0052] Figure 6 A schematic diagram of the layout of a surgical robot system according to an embodiment of this specification is shown;

[0053] Figure 7 A schematic diagram of the layout of a surgical robot system according to an embodiment of this specification is shown;

[0054] Figure 8 Shown Figure 6 A schematic diagram of the layout of the scene scanning device group under the layout shown;

[0055] Figure 9 A schematic diagram showing the establishment of a virtual scene perspective in an embodiment of this specification is shown;

[0056] Figure 10 A schematic diagram showing the contents that can be displayed by the first mixed reality device, the second mixed reality device, and the third mixed reality device in an embodiment of this specification is shown;

[0057] Figure 11 It shows a schematic diagram of system data flow in the field operation mode in one embodiment of this specification;

[0058] Figure 12 It shows a schematic diagram of system data flow in remote operation mode in one embodiment of this specification;

[0059] Figure 13 It shows a schematic diagram of system data flow in the on-site collaboration mode in one embodiment of this specification;

[0060] Figure 14It shows a schematic diagram of system data flow in remote collaboration mode in one embodiment of this specification;

[0061] Figure 15 It shows a schematic diagram of system data flow in evaluation mode in one embodiment of this specification;

[0062] Figure 16 A flow chart showing a data processing method applied to a surgical robot system in an embodiment of this specification is shown;

[0063] Figure 17 A schematic diagram of a data processing device applied to a surgical robot system in an embodiment of this specification is shown;

[0064] Figure 18 A schematic diagram of a computer device in an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0065] The principles and spirit of this specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this specification more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0066] Those skilled in the art will appreciate that the embodiments of this specification may be implemented as a system, device, method, or computer program product. Therefore, the disclosure herein may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0067] This specification provides a surgical robot system. Figure 1 , shows a schematic diagram of the structure of the surgical robot system. Figure 1 As shown, the surgical robot system 10 may include a mixed reality device 101 , a robotic arm 102 , and a data processor 103 .

[0068] The mixed reality device 101 may include a first mixed reality device 111 and a second mixed reality device 112. The first mixed reality device 111 may be worn by a performer and may be provided with a first image acquisition device. The first image acquisition device may be used to acquire first image data from the performer's perspective. The second mixed reality device 112 may be worn by an assistant and may be provided with a second image acquisition device. The second image acquisition device may be used to acquire second image data from the assistant's perspective. In one embodiment, the mixed reality device may be a head-mounted mixed reality pair of glasses. The image acquisition device may be positioned adjacent to the lenses of the head-mounted mixed reality pair of glasses.

[0069] A surgical instrument and a third image acquisition device are mounted at the end of the robotic arm 102. The surgical instrument can perform a surgical operation under the control of the robotic arm 102. The third image acquisition device can acquire third image data from the perspective of the robotic arm end.

[0070] In some embodiments of this specification, the surgical robot system may include a robotic arm dental implant system. Accordingly, the surgical operating instrument is an implant operating instrument, and the surgical operation is a dental implant operation. In some embodiments of this specification, the surgical robot system may also be applied to other surgical procedures in which the robotic arm may interfere with the environment, assistants, or the operating object, such as some orthopedic surgeries, some neurosurgeries, etc. Accordingly, the surgical operating instrument is an orthopedic surgical operating instrument or a neurosurgery operating instrument, and the surgical operation is an orthopedic surgical operation or a neurosurgery operation.

[0071] The data processor 103 can be communicatively connected with the robotic arm 102 and the mixed reality device 101. The data processor 103 can acquire and integrate the first image data, the second image data and the third image data. In one embodiment, the data processor 103 can match the key points in the first image data, the second image data and the third image data to perform transformation and integration between coordinate systems to obtain first integrated data. In one embodiment, the data processor 103 can match the key points in the second image data and the third image data to perform transformation and integration between coordinate systems to obtain second integrated data. The first mixed reality device 111 can acquire and display the first integrated data. The second mixed reality device 112 can acquire and display the second integrated data.

[0072] In the above embodiment, the assistant's perspective and the perspective of the end of the robotic arm can be introduced into the executor's perspective. The perspective of the end of the robotic arm can display the actual situation of the area to be operated on, and the assistant's perspective can supplement the visible range of the executor's perspective, effectively broadening the executor's field of view, so that the executor can promptly discover possible interference between the robotic arm and the surrounding environment during operation, avoid interference between the robotic arm and the environment, assistant or operation object, and facilitate the executor to control the robotic arm to perform surgery, improve the accuracy of the surgery, and improve the surgical experience of the operation object.

[0073] Please continue to refer to Figure 1 ,like Figure 1As shown, in some embodiments of the present specification, the surgical robot system may further include a scene scanning device 104. The scene scanning device 104 may be used to scan the operating object (i.e., the object of the surgical operation, such as a patient) and the environmental information in which the robotic arm 102 is located. The scene scanning device 104 may send the scanned environmental information to the data processor 103. The data processor 103 may establish a corresponding virtual scene based on the environmental information. The first mixed reality device 111 and the second mixed reality device 112 are also used to receive and display the virtual scene. The first mixed reality device 111 may superimpose the virtual scene and the first integrated data for display. The second mixed reality device 112 may superimpose the virtual scene and the second integrated data for display. In this embodiment, the scene scanning device may provide more scene information to the assistant and the executor, so that the executor and the assistant can collaborate to complete the operation.

[0074] In some embodiments of this specification, the scene scanning device 104 may include a natural light camera group and a motion capture camera group. The natural light camera group is used to capture environmental image data. The motion capture camera group is used to capture motion image data of an object containing a recognizable identifier. The recognizable identifier is provided on the first mixed reality device, the second mixed reality device, the robotic arm, and the manipulated object.

[0075] Please refer to Figure 2 , shows a schematic diagram of the principle of a single scene scanning device. The surgical robot system 10 may include at least one scene scanning device. Figure 2 As shown, each scene scanning device includes two camera groups, which scan the operating object and the environment where the robot arm is located to create a corresponding virtual scene. The scenes obtained by different camera groups can be spliced ​​to create the corresponding virtual scene.

[0076] like Figure 2 As shown, the scene scanning device can be integrated with a natural light camera group and a gesture acquisition camera group. In one embodiment, the natural light camera group can include at least two cameras for scanning an environment that changes relatively little over time. The role of the gesture acquisition camera group is to ignore most targets in the environment and only capture the trajectory of the rapid movement of a specified mark or marker, and there is a requirement for the real-time performance of the trajectory of the mark or marker. In one embodiment, the gesture acquisition camera group can include two infrared cameras and an infrared light emitting array, with an array composed of objects that can autonomously emit or passively reflect infrared light as a trackable marker. In another embodiment, the gesture acquisition camera group can include two high-speed cameras, with visual features that can be quickly identified as trackable markers.

[0077] The natural light camera group and the gesture capture camera group can be integrated into a single scene scanning device. Their relative positions are known, and calibration of the two camera groups allows for transformations between the natural light camera group's coordinate system and the gesture capture camera group's coordinate system. The virtual environment scene created using the natural light camera group can be unified with the real-time poses of markers or marker objects captured by the gesture capture camera group in a single coordinate system.

[0078] In some embodiments of this specification, the surgical robot system 10 may integrate multiple scene scanning devices, such as Figure 3 shown. Figure 3 FIG. 1 shows a schematic diagram of the principle of the scene scanning device group. Figure 3 As shown, the scene scanning device group can include two scene scanning devices. The virtual scenes created by different scene scanning devices and the real-time pose information of multiple trackable markers can be unified into a single coordinate system to form the final virtual scene. By configuring multiple scene scanning devices, more scene information can be obtained, thereby providing more scene information to the performer, assistant, and bystanders.

[0079] Please continue to refer to Figure 1 ,like Figure 1 As shown, in some embodiments of this specification, the mixed reality device may further include a third mixed reality device 113. The third mixed reality device 113 may be worn by a bystander. The third mixed reality device 113 may be configured to receive and display the first image data, the second image data, the third image data, and the virtual scene processed by the data processor 103, that is, to overlay and display the virtual scene and the first integrated data.

[0080] Please continue to refer to Figure 1 ,like Figure 1 As shown, in some embodiments of the present specification, the surgical robot system may further include: a force feedback teleoperation device 105. The force feedback device may be suitable for surgical scenarios where surgical instruments are in direct contact with hard tissue. The force feedback teleoperation device 105 may be communicatively connected to the data processor 103, and be used to receive force information during the surgical operation that is sent from the surgical instrument to the data processor 103 via the robotic arm. The force feedback teleoperation device 105 may provide tactile feedback to its user during the surgical operation based on the force information. In one embodiment, the user of the force feedback teleoperation device may be a bystander. In another embodiment, the user of the force feedback teleoperation device may be an executor.

[0081] The force feedback teleoperation device 105 can also be used to capture the operation instructions of its user and send the captured operation instructions to the data processor 103. The data processor 103 can generate a robotic arm execution instruction based on the operation instruction and send the robotic arm execution instruction to the robotic arm 102, so that the robotic arm 102 controls the surgical instrument to perform the surgical operation based on the execution instruction.

[0082] The force feedback teleoperation device 105 can be used for indirect operation of the robotic arm, and can also allow the operator or bystander to intuitively feel the force applied to the surgical instrument associated with the robotic arm during the surgical operation.

[0083] In one embodiment, a bystander can use a force feedback teleoperation device 105 to receive tactile feedback during the surgical operation. The bystander can also obtain visual feedback during the surgical operation by wearing a third mixed reality device 113. In some embodiments, there can be multiple bystanders, and correspondingly, each bystander can wear a corresponding third mixed reality device 113 and use a corresponding feedback teleoperation device 105. The bystander can be an intern, and better learning effects can be achieved through visual feedback and tactile feedback. The bystander can also be an expert, and the operation can be better evaluated through visual feedback and tactile feedback. In the case where the bystander is an expert, the expert can also control the robotic arm 102 through the feedback teleoperation device 105 to assist the executor and assistant in performing the surgical operation. In the above manner, it is convenient for bystanders to learn, evaluate or remotely assist in the operation.

[0084] In another embodiment, a surgeon can use a force feedback teleoperation device 105 to remotely perform a surgical procedure. Specifically, when performing a surgical procedure remotely, the surgeon can wear the first mixed reality device 111 to obtain visual feedback during the surgical procedure. The surgeon can use the force feedback teleoperation device 105 to operate the robotic arm 102, thereby controlling the surgical instruments mounted on the robotic arm 102 to perform the surgical procedure. Furthermore, the surgeon can also obtain tactile feedback through the force feedback teleoperation device. This facilitates the surgeon to perform a surgical procedure remotely.

[0085] In some embodiments of this specification, the surgical procedure is a dental implant procedure, which may include one or more of the following: A reamer provided with the implant may be used to prepare the hole in stages, accompanied by adequate irrigation and cooling, until the implant site is complete. A submerged dental implant may be placed, with the upper edge of the implant aligned with the alveolar crestal bone surface. In another embodiment, a maxillary sinus lift may also be performed.

[0086] The force feedback teleoperation device 105 can be a 6-DOF mechanism consisting of a parallel mechanism and a wrist joint mechanism, or a 6-DOF serial mechanism, which is used to capture the hand movements of the executor or bystander, and map the captured movements into the movement of the robotic arm, while at the same time feeding back to the executor or bystander the force conditions of the surgical operating instruments associated with the robotic arm.

[0087] Please refer to Figure 4 , shows a schematic diagram of a force feedback teleoperation device. Figure 4 As shown, the force feedback teleoperation device 105 is a 6-DOF mechanism consisting of a parallel mechanism and a wrist joint mechanism, including a parallel part, a series part, and a surgical instrument simulation end. The parallel part realizes the translation of the force feedback teleoperation device 105 in three directions, the series part realizes the rotation of the force feedback teleoperation device 105 around three directions, and the surgical instrument simulation end simulates the grip end of the surgical instrument. The force feedback teleoperation device 105 can capture the action of the performer or bystander operating the surgical instrument simulation end and map the captured action to the movement of the robotic arm 102. At the same time, the force feedback teleoperation device 105 can also feedback the force conditions of the surgical instrument associated with the robotic arm to the performer or bystander, providing tactile feedback to the user.

[0088] Please refer to Figure 5 , shows a schematic diagram of the structure of the surgical robot system in the embodiment of this specification. Figure 5 As shown, there are three types of roles in the system wearing mixed reality devices, namely, the performer, the assistant, and the bystander. The performer can wear a first mixed reality device 111 to directly operate the robotic arm 102 or indirectly operate the robotic arm 102 through a force feedback teleoperation device 105. The assistant can wear a second mixed reality device 112 to contact the operation object, cooperate with the performer, or cooperate with the performer to indirectly operate the robotic arm 102 to complete the surgical operation. The bystander can wear a third mixed reality device 113 to feel the operation of the robotic arm 102 through the force feedback teleoperation device 105, or indirectly operate the robotic arm 102 through the force feedback teleoperation device 105.

[0089] The scene scanning device 104 can scan the environment surrounding the manipulated object and the robotic arm 102, transmitting the acquired data to the data processor 103 to create a corresponding virtual scene. The first mixed reality device 111, the second mixed reality device 112, the robotic arm 102, and the manipulated object are all marked with identifiers identifiable by the scene scanning device 104. This allows the position of the operator or assistant, the posture of the robotic arm 102, and the relative position of the manipulated object within the virtual scene to be visible within the virtual scene. The planned manipulated object data can be imported into the data processor 103, allowing it to appear within the virtual scene.

[0090] A surgical instrument is mounted at the end of the robotic arm 102. The force feedback generated by the instrument during surgery is transmitted to the data processor 103 via the robotic arm 102. The posture information of the robotic arm 102 is transmitted to the data processor 103 in the form of parameters for each of the robotic arm's joints. Combined with the robotic arm model stored in the data processor 103, the posture of the robotic arm 102 can be reproduced in real time within the virtual scene. A camera is mounted at the end of the robotic arm 102, which transmits the end-viewing angle of the robotic arm 102 to the data processor 103. The robotic arm 102 can be operated directly by the operator or can move according to execution instructions issued by the data processor 103.

[0091] The force feedback teleoperation device 105 can capture the user's movements, generate operation instructions, and transmit them to the data processor 103 as the basis for the robot arm 102 to execute the instructions. At the same time, the force feedback teleoperation device 105 also provides tactile feedback to the user based on the force feedback information sent by the data processor 103.

[0092] Please continue to refer to Figure 5 ,like Figure 5 As shown, in some embodiments of this specification, the data processor 103 can also be used to receive operation object planning data. The operation object planning data includes operation object image data and / or surgical operation plan data. The data processor 103 can also be used to establish a virtual scene based on the environmental information and the operation object planning data. In this way, the operation object planning data can be displayed in the virtual scene, providing further reference for the surgeon, thereby improving surgical efficiency and enhancing the surgical experience.

[0093] Please continue to refer to Figure 1 In some embodiments of the present specification, the surgical robot system 10 may further include a surgical evaluation device 106. The surgical evaluation device 106 may be configured to obtain the first image data, the second image data, the third image data, the virtual scene, and the force information from the data processor 103. Subsequently, the surgical evaluation device 106 may evaluate the operation corresponding to the surgical operation based on the first image data, the second image data, the third image data, the virtual scene, and the force information. In this manner, a technical evaluation of the operation may be achieved.

[0094] In some embodiments of the present specification, the surgical evaluation device 106 can evaluate the surgery corresponding to the surgical operation based on the trained machine learning model and output a surgical evaluation result. In this way, the efficiency and objectivity of the surgical evaluation can be improved.

[0095] Please refer to Figure 6, shows a schematic diagram of the layout of the surgical robot system in the embodiment of this specification. Figure 6 As shown, the operator, the assistant, and the object of operation can be located in the same surgical site. The operator wears a first mixed reality device 111, and the assistant wears a second mixed reality device 112. The scene scanning device 104 scans the surgical site environment, the robotic arm 102, the object of operation, the first mixed reality device 111, and the second mixed reality device 112 to establish a corresponding virtual scene. The object of operation takes a lying position. The operator directly operates the robotic arm 102 to perform the surgery, and the assistant assists the operator in performing the surgery. The bystander wears a third mixed reality device 113 and is not at the surgical site. Through the force feedback teleoperation device 105, the operator can feel the force feedback information of the surgical instrument when performing the surgery. In the current layout embodiment, the bystander can also indirectly control the robotic arm at the surgical site through the force feedback teleoperation device 105.

[0096] Please refer to Figure 7 , shows a schematic diagram of the layout of a surgical robot system in another embodiment of this specification. Figure 7 As shown, the assistant and the subject are located at the surgical site, while the operator is remote. The operator wears a first mixed reality device 111, and the assistant wears a second mixed reality device 112. The scene scanning device 104 scans the surgical site environment, the robotic arm 102, the subject, and the second mixed reality device 112 to create a corresponding virtual scene. The subject is in a lying position. The remote operator indirectly controls the robotic arm 102 to perform the surgery using a force feedback teleoperation device 105, while the assistant cooperates with the robotic arm 102 to perform the surgery. The spectator wears a third mixed reality device 113 and is not at the surgical site. The spectator uses the force feedback teleoperation device 105 to sense the forces acting on the surgical instruments mounted at the end of the robotic arm 102 during the procedure. In the current embodiment, the spectator can also indirectly control the robotic arm 102 at the surgical site using the force feedback teleoperation device 105.

[0097] Please refer to Figure 8 , showing Figure 6 The layout diagram of the scene scanning device group under the layout shown is as follows. Figure 8 As shown, the operator wears a first mixed reality device 111, which is equipped with a trackable marker. The assistant wears a second mixed reality device 112, which is also equipped with a trackable marker. The operator wears a trackable marker and is positioned near the surgical area. The end and base of the robotic arm 102 are both equipped with trackable markers. The environmental scene information obtained by the scene scanning device 104 and the real-time position information of these trackable markers will serve as the necessary input for establishing the virtual scene.

[0098] Figure 9 Figure 2 shows a schematic diagram of establishing a virtual scene perspective. Figure 9 As shown, the natural light camera group scans the environment to obtain the environmental scene. The rapid motion capture camera group outputs the position and pose information of each trackable marker in real time. Through calibration, the transformation relationship between the coordinate system corresponding to the natural light camera group and the coordinate system of the rapid motion capture camera group can be determined. The trackable markers include the end marker of the robot arm, the robot arm base marker, the operation object marker, the first mixed reality device marker, and the second mixed reality device marker.

[0099] Through robot arm registration, a transformation relationship between the robot arm coordinate system and the fast motion capture camera group coordinate system can be established. One embodiment of robot arm registration can be: based on the robot arm coordinate system, the robot arm is controlled to move to several specified positions. After the robot arm reaches the specified position, the position information of the robot arm end mark and the robot arm base mark in the fast motion capture camera group coordinate system is recorded respectively. Based on this information, the transformation relationship between the robot arm coordinate system and the fast motion capture camera group coordinate system is calculated. The robot arm can output the parameters of each joint in real time. Combined with the known robot arm model parameters, based on the results of the robot arm registration, the real-time position information of the robot arm in the virtual scene can be output.

[0100] One embodiment of subject registration can involve associating a trackable marker worn by the subject with several visible markers in the image data, with the positional relationship between the markers and the trackable markers known. This allows the transformation between the subject's image coordinate system and the coordinate system of the fast motion capture camera set to be derived using the marker information in the image data. Based on the results of subject registration, the pose of the subject's image data and the associated surgical plan within the virtual scene can be output.

[0101] The environmental scene, robotic arm posture, image data of the operated object and surgical plan, and the virtual posture of the mixed reality device are aggregated into the data processor, which ultimately generates virtual scene perspective information that can be observed in the mixed reality device.

[0102] Please refer to Figure 10 , shows a schematic diagram of the contents that can be displayed by the first mixed reality device, the second mixed reality device, and the third mixed reality device in the embodiment of this specification. Figure 10 As shown, the first mixed reality device 111 can superimpose the display assistant perspective, the virtual scene perspective, and the robot arm end effector perspective, wherein the virtual scene perspective includes information such as Figure 9 As shown. Figure 7In the remote scenario shown, the operator can plan the trajectory of the robotic arm 102's end effector into the area near the oral cavity based on the virtual scene perspective, avoiding interference between the robotic arm 102 and the environment, assistant, or object being operated on. The robotic arm 102 end effector perspective can display the actual conditions of the surgical area, and the assistant's perspective can supplement the visible range of the robotic arm 102 end effector perspective. The second mixed reality device 112 can overlay the virtual scene perspective and the robotic arm end effector perspective. The virtual scene perspective can help the assistant detect possible interference with the robotic arm and overcome the limitations of the operator's perspective, ensuring that the operator's field of view is effectively expanded. The third mixed reality device 113 can display the operator's perspective, assistant's perspective, virtual scene perspective, and robotic arm end effector perspective. The first mixed reality device 111, the second mixed reality device 112, and the third mixed reality device 113 can all display various quantitative indicators associated with the robotic arm 102 as needed, such as the relative position of the surgical instruments associated with the robotic arm 102 end effector relative to the object being operated on, and the forces applied during the surgery.

[0103] The following describes the working modes of this embodiment. The surgical robot system in the embodiment of this specification can work in an operation mode, a collaboration mode, and an evaluation mode.

[0104] In the operation mode, the executor can be in the same space with the assistant and the operation object. At this time, the system data flow is as follows: Figure 11 As shown. Figure 11 As shown, the operator wears a first mixed reality device 111, which transmits the operator's perspective information to the data processor 103, and the data processor 103 provides visual feedback to the operator through the first mixed reality device 111. The assistant wears a second mixed reality device 112, which transmits the assistant's perspective information to the data processor 103, and the data processor 103 provides visual feedback to the assistant through the second mixed reality device 112. The scene scanning device 104 provides the data processor 103 with virtual scene perspective information. After the operation object planning data is input into the data processor 103, it also becomes part of the virtual scene perspective. The robotic arm 102 is directly operated by the operator and transmits robotic arm posture information, force feedback information, and end-point perspective information to the data processor 103 in real time. The spectator can obtain visual feedback in real time or non-real time through the third mixed reality device 113 and obtain tactile feedback of the surgical instruments during the operation through the force feedback teleoperation device 105 on the spectator's end. In a non-real-time situation, the tactile feedback provided by the force feedback teleoperation device 105 on the spectator side and the visual feedback provided by the third mixed reality device 113 still need to be synchronized. The system supports more than one spectator receiving the visual feedback and tactile feedback provided by the data processor.

[0105] In the operation mode, the operator can also perform the operation remotely without being in the same space as the assistant or the object. Figure 12 As shown, the basic process and Figure 11 The situation in the field is similar. Figure 12 As shown, the virtual scene perspective only includes the relative positions of the second mixed reality device 112, the operating object, and the robotic arm in the scene. The executor obtains visual feedback remotely, outputs action instructions to the data processor 103 through the force feedback teleoperation device 105, and obtains force feedback information. In this case, the robotic arm 102 performs the corresponding operation after receiving the operation instructions issued by the data processor 103. In the operation mode (remote), the action instructions output by the executor through the force feedback teleoperation device 105, the received visual feedback and tactile feedback must ensure real-time and synchronization with the data generated at the surgical site.

[0106] Please refer to Figure 13 and Figure 14 , respectively showing the data flow diagram in on-site collaboration mode and in remote collaboration mode. Figure 13 and Figure 14 As shown, in collaborative mode, the data streams and operation modes generated by the operator, assistant, robotic arm, and scene scanning device 104 are consistent. The spectator needs to obtain real-time visual feedback from the data processor 103 via the third mixed reality device 113. The spectator receives tactile feedback via the force feedback teleoperation device 105. If the spectator determines that the operator's operation, whether on-site or remote, has problems or potential risks, they can communicate with the operator and assistant through the data processor 103 to provide guidance. Alternatively, they can take over the operator's role based on the actual situation at the surgical site, operating the force feedback teleoperation device 105 and outputting motion instructions to the data processor 103. The data processor 103 then outputs robotic arm operation instructions, enabling the spectator to indirectly operate the robotic arm. The assistant then cooperates with the robotic arm 102, which is indirectly operated by the spectator, to perform the surgery. At this point, the motion instructions output by the spectator via the force feedback teleoperation device 105 and the visual and tactile feedback received from the data processor 103 must be synchronized and in real time with the surgical scene.

[0107] Please refer to Figure 15 , respectively show the data flow diagram in the evaluation mode. Figure 15As shown, in the evaluation mode, the data streams generated by the executor, assistant, robotic arm 102, and scene scanning device 104 are consistent with the operation mode. The onlooker can evaluate the operations of the evaluated object, the executor, and the assistant in real time or non-real time based on the visual feedback and tactile feedback provided by the data processor. During the bystander evaluation process, the perspective information (i.e., visual feedback) needs to be synchronized with the force feedback information (i.e., tactile feedback); the bystander can evaluate the rationality of the executor and assistant's operations based on the various feedback information obtained. The bystander performing the evaluation can be a medical expert or an AI model, wherein the AI ​​model does not need the third mixed reality device 113 and the force feedback teleoperation device 105 when performing the evaluation, and can directly perform the evaluation based on the data provided by the data processor.

[0108] The surgical robot system in the above-mentioned embodiment solves the limitation of surgical field of view by providing multiple perspectives; the robotic arm can not only ensure the accuracy of surgical operation, but also provide the stress conditions of surgical instruments during operation; by considering the needs of intraoperative assistants and scanning of the environment, the system's ability to cope with complex scenes is improved; the bystander perspective can meet the user's learning needs, and can also meet the needs of evaluating the user.

[0109] The embodiments of this specification provide a data processing method applied to a surgical robot system. Figure 16 A flow chart of a data processing method applied to a surgical robot system in an embodiment of this specification is shown. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures that do not logically have a necessary causal relationship, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of this specification and shown in the drawings. When the method or module structure described is applied to an actual device or terminal product, it can be connected in accordance with the method or module structure shown in the embodiments or drawings for sequential execution or parallel execution (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0110] Specifically, if Figure 16 As shown, a data processing method for a surgical robot system provided in one embodiment of this specification may include the following steps:

[0111] Step S161, obtaining first image data, second image data and third image data; the first image data is image data collected from the executor's perspective; the second image data is image data collected from the assistant's perspective; the third image data is image data collected from the end perspective of the robotic arm.

[0112] The method in this embodiment can be applied to a data processor in a surgical robot system.

[0113] The data processor may acquire first image data from the perspective of the performer. The first image data may be acquired by a first image acquisition device in a first mixed reality device worn by the performer. The data processor may also acquire second image data from the perspective of an assistant. The second image data may be acquired by a second image acquisition device in a second mixed reality device worn by the assistant. The data processor may also acquire third image data from the perspective of the robotic arm. The third image data may be acquired by a third image acquisition device mounted at the end of the robotic arm.

[0114] Step S162: Integrate the first image data, the second image data, and the third image data to obtain first integrated data; and integrate the second image data and the third image data to obtain second integrated data.

[0115] After obtaining the first image data, the second image data, and the third image data, the data processor may integrate the first image data, the second image data, and the third image data to obtain first integrated data. In one embodiment, the data processor may align the first image data, the second image data, and the third image data based on an object that exists in the first image data, the second image data, and the third image data, and unify them into a coordinate system to obtain the first integrated data. In another embodiment, the data processor may set the display position of the first image data, the second image data, and the third image data. For example, the first image data may be set to occupy the main part of the displayable area, and the second image data and the third image data may be superimposed on the first image data display area in the form of dynamic thumbnail views.

[0116] The data processor may also integrate the second image data and the third image data to obtain second integrated data. In one embodiment, the data processor may align the second image data and the third image data based on an object present in both the second image data and the third image data, unifying them into a single coordinate system to obtain the second integrated data. In another embodiment, the data processor may set the display positions of the second image data and the third image data. For example, the second image data may be set to occupy the majority of the displayable area, and the third image data may be superimposed on the second image data display area in the form of a dynamic thumbnail view.

[0117] Step S163: Send the first integrated data to a first mixed reality device worn by the performer for display; and send the second integrated data to a second mixed reality device worn by the assistant for display.

[0118] The data processor may send the first integrated data to a first mixed reality device worn by the performer for display. The performer may initiate a switch view instruction through the first mixed reality device, and the performer may switch the content in the main display area to the aligned image data, the first image data, the second image data, or the third image data. The data processor may send the second integrated data to a second mixed reality device worn by the performer for display. The assistant may initiate a switch view instruction through the second mixed reality device, switch to different perspectives as needed, and observe the aligned second image data and the third image data.

[0119] In the above embodiment, the assistant's perspective and the perspective of the end of the robotic arm can be introduced into the executor's perspective. The perspective of the end of the robotic arm can display the actual situation of the area to be operated on, and the assistant's perspective can supplement the visible range of the executor's perspective, effectively broadening the executor's field of view, so that the executor can promptly discover possible interference between the robotic arm and the surrounding environment during operation, avoid interference between the robotic arm and the environment, assistant or operation object, and facilitate the executor to control the robotic arm to perform surgery, improve the accuracy of the surgery, and improve the surgical experience of the operation object.

[0120] In some embodiments of the present specification, the data processing method further includes: obtaining force information of the surgical instrument during the surgical operation; sending the force information to a force feedback remote operation device, so that the force feedback remote operation device provides tactile feedback to the user during the surgical operation based on the force information.

[0121] Specifically, the force information received by a force feedback teleoperation device during surgery is spatially directional. When the user grips the device, it provides tactile feedback in the same direction, based on the direction and magnitude of the force applied. When used in training or assessment, combined with force feedback information, this can provide a more comprehensive learning experience or assessment recommendations.

[0122] In some embodiments of the present specification, the data processing method further includes: receiving an operation instruction sent by a force feedback remote operation device; generating a robotic arm execution instruction based on the operation instruction, and sending the robotic arm execution instruction to the robotic arm.

[0123] Specifically, the data processor can receive operating instructions sent by a force feedback teleoperation device, which may include position and posture data of the force feedback teleoperation device at one or more moments. The data processor can then generate robotic arm execution instructions based on the operating instructions and send them to the robotic arm. In one embodiment, after a user operates the force feedback teleoperation device, the system records the position and posture of the force feedback teleoperation device at each moment during the operation at fixed intervals. The data processor can receive the position and posture data of the force feedback teleoperation device at each moment in real time. Based on the change in the position and posture data of the force feedback teleoperation device between the current moment and the previous moment, the data processor can determine the corresponding change in the robotic arm's position and posture at the current moment. Since the fixed time interval is known, the system can determine the movement speed of the robotic arm based on the change; based on the change in position and posture and the movement speed, the system can generate robotic arm execution instructions. In this way, surgery can be performed remotely using a force feedback teleoperation device.

[0124] In some embodiments of the present specification, the data processing method also includes: obtaining environmental information of the environment in which the operating object and the robotic arm are located and operating object planning data; the operating object planning data includes operating object image data and / or surgical operation plan data; establishing a corresponding virtual scene based on the environmental information and the operating object planning data, and sending the virtual scene to the first mixed reality device and the second mixed reality device for display.

[0125] Specifically, the environmental information of the environment in which the operating object and the robotic arm are located can be scene image data collected by the scene scanning device. The scene scanning device may include multiple cameras, each used to collect environmental image data that changes relatively little over time and motion image data that changes quickly over time. The motion image data here may include motion image data corresponding to the robotic arm, the operating object, the first mixed reality device and the second mixed reality device. The data processor can establish a corresponding virtual scene based on these data and the operating object planning data. The operating object planning data may include operating object image data and / or surgical operation plan data. The data processor can also send the established virtual scene to the first mixed reality device and the second mixed display device for display. The first mixed reality device can superimpose the virtual scene and the first integrated data for display. The second mixed reality device can superimpose the virtual scene and the second integrated data for display. In this embodiment, by constructing and displaying a virtual scene, more scene information can be provided to the assistant and the executor, so that the executor and the assistant can collaborate to complete the operation.

[0126] In some embodiments of the present specification, the environmental information includes environmental image data and motion image data, and the motion image data includes posture information of an object corresponding to each identifiable identifier in a plurality of identifiable identifiers; the plurality of identifiable identifiers include: a robotic arm identifier, an operation object identifier, a first mixed reality device identifier, and a second mixed reality device identifier; accordingly, a corresponding virtual scene is established according to the environmental information and the operation object planning data, including: determining a first transformation relationship between a coordinate system corresponding to the environmental image data and a coordinate system corresponding to the motion image data through calibration; establishing a second transformation relationship between a coordinate system of the robotic arm and a coordinate system of the motion image data through robotic arm registration; establishing a third transformation relationship between a coordinate system of the operation object and a coordinate system of the motion image data through operation object registration; establishing a fourth transformation relationship between a coordinate system of the first mixed reality device and a coordinate system of the motion image data through first mixed reality device registration; establishing a fifth transformation relationship between a coordinate system of the second mixed reality device and a coordinate system of the motion image data through second mixed reality device registration; and establishing a corresponding virtual scene based on the environmental image data, the motion image data, the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, the fifth transformation relationship, and the operation object planning data.

[0127] Specifically, the operator wears a first mixed reality device with a trackable marker attached. The assistant wears a second mixed reality device, also with a trackable marker attached. The subject wears a trackable marker and is positioned near the area to be operated on. The end and base of the robotic arm both carry trackable markers. The environmental scene information obtained by the scene scanning device and the real-time position information of these trackable markers will serve as necessary inputs for establishing the virtual scene. The environmental scene, robotic arm posture, operating subject image data and surgical plan, and the virtual posture of the mixed reality device are aggregated into a data processor, ultimately generating virtual scene perspective information that can be observed in the mixed reality device. The environmental information includes environmental image data and motion image data. The motion image data includes the position information of objects corresponding to each of a plurality of identifiable identifiers. The environmental image data includes image data of objects whose positions in the scene change relatively little. Through camera calibration, the coordinate system transformation relationship between the environmental image data and the motion image data can be obtained. Through robotic arm registration, the coordinate system transformation relationship between the robotic arm coordinate system and the coordinate system of the motion image data can be determined. By registering the operating object, a coordinate transformation relationship between the coordinate system of the operating object and the motion image data can be established. By registering the first mixing device, a coordinate transformation relationship between the first mixing device and the motion image data can be established. By registering the second mixing device, a coordinate transformation relationship between the second mixing device and the motion image data can be established. Subsequently, based on the above coordinate transformation relationship, the environmental image data and the motion image data are aligned to obtain virtual image data. The operating object planning data is then superimposed and displayed on the virtual image data to obtain a virtual scene.

[0128] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the aforementioned related processing embodiments, and no further description is given here.

[0129] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] Based on the same inventive concept, the embodiments of this specification also provide a data processing device applied to a surgical robot system, as described in the following embodiments. Since the principle of solving the problem by the data processing device applied to the surgical robot system is similar to the data processing method applied to the surgical robot system, the implementation of the data processing device applied to the surgical robot system can refer to the implementation of the data processing method applied to the surgical robot system, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. Figure 17 This is a structural block diagram of a data processing device applied to a surgical robot system according to an embodiment of this specification. Figure 17 As shown, it includes: an acquisition module 171, a processing module 172 and a sending module 173, and the structure is described below.

[0131] The acquisition module 171 is used to acquire first image data, second image data and third image data; the first image data is image data collected from the executor's perspective; the second image data is image data collected from the assistant's perspective; and the third image data is image data collected from the end perspective of the robotic arm.

[0132] The processing module 172 is configured to integrate the first image data, the second image data, and the third image data to obtain first integrated data; and integrate the second image data and the third image data to obtain second integrated data.

[0133] The sending module 173 is used to send the first integrated data to the first mixed reality device worn by the performer for display; and send the second integrated data to the second mixed reality device worn by the assistant for display.

[0134] This specification also provides a medical device. Figure 18 The diagram shows the structure of a computer device that is used in a data processing method for a surgical robot system according to an embodiment of this specification. The computer device may include an input device 181, a processor 182, and a memory 183. The memory 183 is used to store processor-executable instructions. When the processor 182 executes these instructions, the steps of the data processing method for a surgical robot system described in any of the above embodiments are implemented.

[0135] In this embodiment, the input device can specifically be one of the primary devices for exchanging information between a user and a computer system. The input device can include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing these data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple levels. In digital systems, anything that can store binary data can be considered a memory device. In integrated circuits, a circuit with storage functionality that does not have a physical form is also called a memory device, such as a RAM or FIFO. In systems, a physical storage device is also called a memory device, such as a memory stick or a TF card.

[0136] In this embodiment, the specific functions and effects achieved by the computer device can be explained in comparison with other embodiments and will not be repeated here.

[0137] In an embodiment of this specification, a computer storage medium based on a data processing method applied to a surgical robot system is also provided, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the data processing method applied to a surgical robot system described in any of the above embodiments are implemented.

[0138] In this embodiment, the storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured in accordance with the standards specified by the communication protocol for network connection communication.

[0139] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0140] Obviously, those skilled in the art should understand that the various modules or steps of the above-mentioned embodiments of this specification can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the embodiments of this specification are not limited to any specific combination of hardware and software.

[0141] It should be understood that the above description is intended to be illustrative and not limiting. Numerous embodiments and applications beyond the examples provided will be readily apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined with reference to the above description, but rather with reference to the preceding claims, along with the full scope of equivalents to which such claims are entitled.

[0142] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Those skilled in the art will readily appreciate that various modifications and variations to the embodiments of this specification are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be within the scope of protection of this specification.

Claims

1. A surgical robot system, characterized in that: include: The mixed reality device includes a first mixed reality device and a second mixed reality device; the first mixed reality device is worn by a performer and is provided with a first image acquisition device for acquiring first image data from the performer's perspective; The second mixed reality device is worn by the assistant and is provided with a second image acquisition device for acquiring second image data from the assistant's perspective; A robotic arm, wherein a surgical instrument and a third image acquisition device are mounted at the distal end of the robotic arm; the surgical instrument is used to perform a surgical operation under the control of the robotic arm; and the third image acquisition device is used to acquire third image data from the perspective of the distal end of the robotic arm; a data processor, communicatively connected to the robotic arm and the mixed reality device, configured to acquire and integrate the first image data, the second image data, and the third image data; The first mixed reality device is used to acquire and display the first image data, the second image data and the third image data processed by the data processor; the second mixed reality device is used to acquire and display the second image data and the third image data processed by the data processor.

2. The surgical robot system according to claim 1, wherein: Also includes: A force feedback teleoperation device is communicatively connected to the data processor and is used to receive force information during a surgical operation sent by the surgical instrument to the data processor via the robotic arm, so as to provide the user with tactile feedback during the surgical operation based on the force information; the force feedback teleoperation device is also used to capture the user's operating instructions and send the captured operating instructions to the data processor, so that the data processor generates a robotic arm execution instruction based on the operating instruction and sends the robotic arm execution instruction to the robotic arm.

3. The surgical robot system according to claim 2, wherein: Also includes: A scene scanning device is used to scan the environmental information of the operating object and the robotic arm, and send the scanned environmental information to the data processor, so that the data processor establishes a corresponding virtual scene based on the environmental information; the first mixed reality device and the second mixed reality device are also used to receive and display the virtual scene.

4. The surgical robot system according to claim 3, wherein: The data processor is further configured to receive operation object planning data; the operation object planning data includes operation object image data and / or surgical operation plan data; the data processor is further configured to establish a virtual scene based on the environmental information and the operation object planning data.

5. The surgical robot system according to claim 3, wherein: The scene scanning device includes a natural light camera group and a motion capture camera group; The natural light camera group is used to collect environmental image data; The motion capture camera group is used to collect motion image data of an object containing a identifiable identifier; the first mixed reality device, the second mixed reality device, the robotic arm and the operating object are all provided with the identifiable identifier.

6. The surgical robot system according to claim 3, wherein: The mixed reality device also includes a third mixed reality device worn by a bystander, and the third mixed reality device is used to receive and display the first image data, the second image data, the third image data and the virtual scene processed by the data processor.

7. The surgical robot system according to claim 3, wherein: Also includes: A surgical evaluation device is used to obtain the first image data, the second image data, the third image data, the virtual scene and the force information from the data processor, and evaluate the surgery corresponding to the surgical operation based on the first image data, the second image data, the third image data, the virtual scene and the force information.

8. A data processing method applied to a surgical robot system, characterized in that: include: acquiring first image data, second image data, and third image data; The first image data is image data collected from the perspective of the executor; The second image data is image data collected from the assistant's perspective; the third image data is image data collected from the end perspective of the robotic arm; integrating the first image data, the second image data, and the third image data to obtain first integrated data; integrating the second image data and the third image data to obtain second integrated data; sending the first integrated data to a first mixed reality device worn by a performer for display; The second integrated data is sent to a second mixed reality device worn by the assistant for display.

9. The data processing method according to claim 8, characterized in that: Also includes: Obtaining force information of surgical instruments during surgical operations; sending the force information to a force feedback teleoperation device, so that the force feedback teleoperation device provides tactile feedback to the user during the surgical operation based on the force information; receiving operation instructions sent by the force feedback teleoperation device; A robot arm execution instruction is generated based on the operation instruction, and the robot arm execution instruction is sent to the robot arm.

10. The data processing method according to claim 8, characterized in that: Also includes: Obtaining environmental information of the operating object and the environment in which the robotic arm is located and operating object planning data; The operation object planning data includes operation object image data and / or surgical operation plan data; A corresponding virtual scene is established according to the environmental information and the operation object planning data, and the virtual scene is sent to the first mixed reality device and the second mixed reality device for display.

11. The data processing method according to claim 10, characterized in that: The environmental information includes environmental image data and motion image data, wherein the motion image data includes position information of each identifiable marker among a plurality of identifiable markers; The multiple identifiable identifiers include: a robotic arm identifier, an operation object identifier, a first mixed reality device identifier, and a second mixed reality device identifier; Accordingly, establishing a corresponding virtual scene according to the environmental information and the operation object planning data includes: determining, by calibration, a first transformation relationship between a coordinate system corresponding to the environmental image data and a coordinate system corresponding to the motion image data; A second transformation relationship is established between the coordinate system of the robotic arm and the coordinate system of the motion image data by registering the robotic arm; a third transformation relationship is established between the coordinate system of the operating object and the coordinate system of the motion image data by registering the operating object; a fourth transformation relationship is established between the coordinate system of the first mixed reality device and the coordinate system of the motion image data by registering the first mixed reality device; and a fifth transformation relationship is established between the coordinate system of the second mixed reality device and the coordinate system of the motion image data by registering the second mixed reality device. A corresponding virtual scene is established based on the environmental image data, the motion image data, the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, the fifth transformation relationship and the operation object planning data.

12. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the method according to any one of claims 8 to 11 are implemented.

Images