Simulation system, information processing apparatus, and information processing method
Through all-optical communication network and remote DMA technology, the insufficient computing power and delay problems in surgical simulation are solved, and the simulation performance with high real-time and low latency is achieved, which improves the authenticity and effect of surgical training.
Patent Information
- Application Number
- CN202380024755.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-08
- Filing Date
- 2023-02-27
- Publication Date
- 2024-11-12
AI Technical Summary
In surgical simulation, the prior art is difficult to deal with the problems of high real-time and low latency, especially when it involves the presentation of force, resulting in insufficient computing power, poor simulation performance, and difficult to accurately reflect the deformation of large human models. and feedback on force.
By using all-optical communication networks and remote DMA technology, low-latency data transmission and distributed processing between information processing devices are achieved, sufficient computing resources and synchronous data processing are ensured, and surgical simulations are provided in collaboration, including computing soft tissue deformation and force. Feedback.
Real-time processing of large human models and precise force feedback are achieved, simulation performance and training authenticity are improved, surgical results and doctor training quality are improved.
Smart Images

Figure CN118946326A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a simulation system, an information processing device, and an information processing method. Background Art
[0002] In the field of surgery, it is known to perform simulation of a surgical operation (for example, see Patent Document 1).
[0003] Reference List
[0004] Patent Literature
[0005] Patent Document 1: JP 2020-62494A Summary of the invention
[0006] Technical issues
[0007] It is conceivable to use the high computing power of cloud computing to improve simulation performance. In this case, it is necessary to deal with the delay of communication. In particular, in the case of simulations with high real-time nature involving sense-of-force presentation, the problem may become apparent.
[0008] One aspect of the present disclosure improves simulation performance.
[0009] Solution to the problem
[0010] A simulation system according to one aspect of the present disclosure includes: a first information processing device and a second information processing device that provide a simulation of a surgical operation in cooperation with each other, wherein the first information processing device and the second information processing device transmit and receive simulation data to each other via an all-optical communication network.
[0011] According to one aspect of the present disclosure, an information processing device cooperates with another information processing device to provide a simulation of a surgical operation, and the information processing device includes: a processing unit that calculates a simulation model including soft tissue; and an optical transmission control unit that controls the sending of simulation data to the other information processing device and the receiving of simulation data from the other information processing device via an all-optical communication network.
[0012] An information processing method according to one aspect of the present disclosure is for providing a simulation of a surgical operation by cooperation between a first information processing device and a second information processing device, the information processing method comprising: the first information processing device and the second information processing device sending and receiving simulation data to each other via an all-optical communication network. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 is a diagram schematically showing an overview of a simulation system according to an embodiment.
[0014] Figure 2 is a diagram showing an example of functional blocks of a simulation system.
[0015] Figure 3 is a block diagram illustrating an example of a schematic configuration of a haptic device.
[0016] Figure 4 is a diagram showing an example of a video.
[0017] Figure 5 is a diagram showing an example of a simulation cycle.
[0018] Figure 6 is a diagram schematically illustrating remote DMA.
[0019] Figure 7 is a flowchart showing an example of processing (information processing method) performed in the simulation system.
[0020] Figure 8 is a diagram showing a modified schematic configuration.
[0021] Fig. 9 It is a diagram schematically showing the flow of simulation.
[0022] Fig.10 is a diagram showing a modified schematic configuration.
[0023] Fig.11 is a diagram showing an example of the hardware configuration of a device. DETAILED DESCRIPTION
[0024] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in the embodiments described below, redundant descriptions are omitted by denoting the same elements with the same reference numerals and symbols.
[0025] The present disclosure is explained according to the order of the items described below.
[0026] 0. Introduction
[0027] 1. Implementation Method
[0028] 2. Modification
[0029] 3. Hardware Configuration Example
[0030] 4. Effect examples
[0031] 0. Introduction
[0032] Surgery requires a high degree of skill from doctors, and training is important. In conventional surgical simulations, a video display of a simulated organ can be performed, and surgery for that organ can be learned. However, surgical simulations cannot accurately reflect deformation, and the force sense of an actual flexible organ is only used for initial training. Due to the difficulty in handling large human models, surgical simulations are limited to model simulations for some model organs.
[0033] For example, from the perspective of computing resources, it is difficult to handle a large model that includes the entire abdominal organs. In order to perform accurate simulations, it is necessary to make the mesh of the model denser. From the perspective of computing resources, this also has limitations. In order to ensure computing resources, it is conceivable to use the high computing power of cloud computing. However, there is a problem of communication delay. In particular, in the case of real-time simulation, this problem becomes apparent. It is also conceivable to achieve force estimation and feedback that matches physical parameters (such as Young's modulus) by combining FEM (finite element method). However, for more precise tactile control / force feedback, it is necessary to shorten the cycle of arithmetic operations used for simulation and feedback.
[0034] In particular, due to the problem of delay described above, simulation has been performed only by edge computing. Since sufficient computing power cannot be obtained, model simplification and limitation of the modeled target part must be performed, and authenticity is sacrificed. The cycle of arithmetic operation needs to be set longer, and the authenticity of force feedback is also sacrificed. In a long arithmetic operation cycle, it is difficult to realistically simulate the deformation of an organ to which real gravity is applied, etc.
[0035] At least one of the problems described above is solved by the disclosed technology. By using an all-optical communication network, the problem of low latency is solved. Sufficient computing resources through cloud computing are ensured. By using remote DMA (direct memory access), in addition to low-latency data transfer, distributed processing across the network can be easily performed by synchronizing data present in the memory of distributed information processing devices.
[0036] For example, by using high computing power, real-time simulation of deformation of large and flexible human organ models and force sense calculation can be performed. Both real-time processing of large human models and real-time feedback of precise force sense can be achieved. A simulation environment similar to actual surgery is constructed, which is also based on the doctor's delicate sense. It can be expected that the quality of doctors' training and preoperative planning will be improved, thereby improving surgical results.
[0037] 1. Implementation Method
[0038] Figure 11 is a diagram schematically showing an overview of a simulation system according to an embodiment. The simulation system 100 can be used as a tactile surgical simulator capable of performing large-scale arithmetic operations. For example, the simulation system 100 can be used for training or preoperative planning of surgical operations (e.g., laparoscopic surgery, thoracic surgery, and brain surgery) that process soft tissues (soft organs, etc.). By training while feeling the deformation and real reaction of soft tissues, more accurate prior experience can be performed, and effects in terms of improvement of skills and improvement of surgical results can be expected.
[0039] Note that the simulation system 100 can also be used as a training system for surgical robots. Quite a few conventional surgical support robots do not have force sense control. In this case, the doctor performs treatment while only imagining the force applied to the organ or tissue using video. By calculating and presenting (e.g., displaying) an accurate sense of force, the force applied to the organ and the appropriate force when suturing can be grasped. More effective training can be performed.
[0040] exist Figure 1 In the illustrated example, the simulation system 100 includes an information processing apparatus 1 and an information processing apparatus 2. The information processing apparatus 1 and the information processing apparatus 2 are a first information processing apparatus and a second information processing apparatus that provide a surgical simulation by cooperating with each other.
[0041] The information processing device 1 and the information processing device 2 are connected via an all-optical communication network N. As described in detail below, the information processing device 1 and the information processing device 2 transmit and receive analog data to and from each other via the all-optical communication network N.
[0042] The connection between the information processing device 1 and the information processing device 2 through the all-optical communication network N means that all communications between at least the endpoint routers are composed of optical communications. Note that the communication after the endpoint routers can be electrical communication after photoelectric conversion. By communicating via the broadband all-optical communication network N, the information processing device 1 and the information processing device 2 are connected with low latency. This is particularly useful for real-time simulation.
[0043] The information processing device 1 is set in the edge area, and constitutes an edge simulator terminal in combination with the tactile device 6 and the monitor 7. In this example, the tactile device 6 and the monitor 7 are also components of the information processing device 1. However, the tactile device 6 and the monitor 7 may not be components of the information processing device 1. The user U of the simulation system 100 is located in the edge area EF. Figure 1 , only a part of the hand of the user U operating the haptic device 6 is schematically shown.
[0044] The surgical operation environment OE is simulated by the cooperation of the information processing device 1 and the information processing device 2. Figure 1In FIG. 1 , a surgical environment OE is schematically shown inside the information processing device 1. The surgical environment OE includes virtual elements related to surgery. As elements included in the surgical environment OE, Figure 1 Soft tissue O, surgical instruments T, robotic arms R and camera devices C are shown in FIG.
[0045] Examples of the soft tissue O include organs having flexibility. Examples of the surgical instrument T are surgical instruments such as forceps. Unlike the soft tissue O, the surgical instrument T may have rigidity. The robot arm R is a surgical robot arm that supports the camera C. The camera C is a camera used for surgery, and images a surgical area including the soft tissue O and the surgical instrument T. The field of view of the camera C is referred to as a field of view F and is shown.
[0046] The haptic device 6 (tactile device) is a device that the user U touches and operates (with his hands) in order to operate the surgical instrument T. The user U moves the haptic device 6 to operate the surgical instrument T. By holding and cutting the soft tissue O using the surgical instrument T, surgical simulation is advanced. When the surgical instrument T contacts the soft tissue O, an external force is applied to the soft tissue O, and the soft tissue O is simulated as, for example, deformed. The contact force generated by the interaction between the soft tissue O and the surgical instrument T (which can be equivalent to, for example, a sense of force) is reflected in the operation of the surgical instrument T by the user U via the haptic device 6, and is fed back to the user U.
[0047] The monitor 7 displays the video in the field of view F of the camera C described above, that is, the video including the soft tissue O and (at least a part of) the surgical instrument T. The user U operates the haptic device 6 while viewing the video displayed on the monitor 7. The position, angle, etc. of the camera C can also be operated by the user U. The operation unit of the camera C can be incorporated into the haptic device 6, or can be provided separately from the haptic device 6. The operation of the camera C can be performed automatically. For example, the function of the control unit that automatically operates the camera C can be incorporated into the surgical environment OE.
[0048] The information processing device 2 is provided in the cloud region CF. The cloud region CF is a region located on the opposite side of the edge region EF across the all-optical communication network N, and is, for example, a region far from the edge region EF.
[0049] Reference Figure 2 A more specific configuration of the simulation system 100 is described.
[0050] Figure 2 is a diagram showing an example of functional blocks of a simulation system. Figure 2 , examples of functional blocks of the information processing device 1 and the information processing device 2 are shown.
[0051] The information processing apparatus 1 includes a processing unit 11 , a storage unit 12 , an IF unit 13 , a communication control unit 14 , an optical transmission control unit 15 , and an optical transmission device 16 in addition to the haptic device 6 and the monitor 7 described above.
[0052] The processing unit 11 functions as an overall control unit (main control unit) that controls the elements of the information processing apparatus 1. The processing unit 11 performs processing related to simulation. The details are explained below.
[0053] The storage unit 12 stores information used in the information processing device 1. As the information stored in the storage unit 12, simulation data is exemplified. The simulation data includes data used in arithmetic operations of simulations and the like and data obtained by simulation.
[0054] The IF unit 13 provides an interface between the processing unit 11 and the haptic device 6 and the monitor 7. Information about position and force (motion) is input from the haptic device 6 to the processing unit 11 via the IF unit 13. The motion may also include information such as posture. The processing unit 11 causes the motion of the robot arm R in the surgical environment OE to correspond to the motion input via the haptic device 6. The user U operates the surgical instrument T in the surgical environment OE via the haptic device 6.
[0055] The contact force generated by the contact of the surgical instrument T with the soft tissue O is calculated. The calculated contact force is fed back to the user U via the haptic device 6 . A command value for reflecting the contact force on the haptic device 6 is sent from the processing unit 11 to the haptic device 6 via the IF unit 13 .
[0056] As explained above, the surgical environment OE includes the camera C, and the video of the camera C is displayed by the monitor 7 ( Figure 1 ). The video of the camera C is sent from the processing unit 11 to the monitor 7 via the IF unit 13 and is displayed.
[0057] The communication control unit 14 controls communication between the information processing apparatus 1 and an external device. An example of the external device is the information processing apparatus 2. In this case, an optical transmission control unit 15 and an optical transmission device 16 described below are used.
[0058] The optical transmission control unit 15 controls optical transmission by the optical transmission device 16. The optical transmission device 16 is a device for connecting the information processing apparatus 1 to the all-optical communication network N. The optical transmission device 16 is mounted on, for example, PCIe (Peripheral Component Interconnect Express) or the like, and is incorporated in the information processing apparatus 1.
[0059] The information processing apparatus 2 includes a processing unit 21 , a storage unit 22 , a communication control unit 24 , an optical transmission control unit 25 , and an optical transmission device 26 .
[0060] The processing unit 21 functions as an overall control unit that controls the elements of the information processing device 2. The processing unit 21 performs processing related to simulation. The details are explained below.
[0061] The storage unit 22 stores information used in the information processing device 2. As the information stored in the storage unit 22, simulation data is exemplified.
[0062] The communication control unit 24 controls communication between the information processing apparatus 1 and an external device. An example of the external device is the information processing apparatus 1. In this case, the optical transmission control unit 25 and the optical transmission device 26 are used.
[0063] The optical transmission control unit 25 controls optical transmission by the optical transmission device 26. The optical transmission device 26 is a device for connecting the information processing apparatus 2 to the all-optical communication network N.
[0064] Reference Figure 3 A specific configuration of the haptic device 6 is explained.
[0065] Figure 3 is a block diagram showing an example of a schematic configuration of a haptic device. The haptic device 6 includes a plurality of (n) joints 61 and a haptic control unit 62. The joints 61 are referred to as a joint 61-1 and a joint 61-n, and are shown to be distinguishable from each other.
[0066] The joint 61 includes a motor 611, an encoder 612, a sensor 613, a motor control unit 614, and a driver 615. The motor 611 rotates according to the current driven by the driver 615. The encoder 612 detects the rotational position of the joint 61. Examples of the sensor 613 are a force sensor, a torque sensor, an acceleration sensor, etc. The sensor 613 detects the force and torque applied to the joint 61, the acceleration of the joint 61, etc. The sensor may be a six-axis sensor, or a plurality of sensors may be combined and used as the sensor. More precise force control may be performed.
[0067] The motor control unit 614 performs feedback control on the motor 611 based on the detection value of the encoder 612 and the detection value of the sensor 613. For example, the position, force, torque, acceleration, etc. of the joint 61 are controlled. The feedback control may include arithmetic operations of force control and acceleration control, including estimation of a disturbance observer. The motor 611 is controlled via the driver 615. The motor control unit 614 provides a current command to the driver 615. The driver 615 drives the current of the motor 611 based on the provided current command.
[0068] The motor control unit 614 sends sensor information, such as the detection value of the encoder 612 and the detection value of the sensor 613 , to the haptic control unit 62 .
[0069] The tactile control unit 62 sends a signal to the processing unit 11 ( Figure 2 )Send position / force information.
[0070] The tactile control unit 62 is based on the data from the processing unit 11 ( Figure 2 ), sends a control command to the motor control unit 614 of the joint 61. The motor control unit 614 of the joint 61 controls the motor 611 based on the control command from the tactile control unit 62.
[0071] Refer again Figure 2 The simulation system 100 is further described. In the simulation system 100, the processing required for the simulation is shared and undertaken by the processing unit 11 of the information processing device 1 and the processing unit 21 of the information processing device 2.
[0072] The processing unit 21 of the information processing device 2 set in the cloud area CF has a higher computing power than the processing unit 11 of the information processing device 1 set in the edge area EF. Among various processes required for the simulation, a process with a particularly large computing load is performed by the processing unit 21 of the information processing device 2. An example of this process is described.
[0073] The processing unit 11 of the information processing apparatus 1 simulates the movement of the surgical instrument T. Position / force information is input to the processing unit 11 from the haptic device 6. Since the surgical instrument T has rigidity and does not deform like the soft tissue O, the calculation load is not large.
[0074] The processing unit 21 of the information processing device 2 calculates a simulation model including the soft tissue O. The calculation of the simulation model includes modeling of the soft tissue O and calculation of deformation of the soft tissue O. For example, FEM, material point method (MPM), etc. can be used. Although the calculation load is large, since the processing unit 21 of the information processing device 2 has high calculation power, the calculation of the simulation model with high accuracy can be performed at high speed.
[0075] In the simulation of the soft tissue O, the XPBD (Extended Position Based Dynamics) method can be used in combination with the FEM. For example, based on the distance between the multiple particles in the polyhedron and the mass of the multiple particles, the deformation of the soft tissue O to which the external force is applied is calculated with the condition that minimizes the potential energy density of the polyhedron modeled based on the FEM as a constraint. This constraint is equivalent to the constraint in the XPBD.
[0076] The processing unit 21 of the information processing device 2 calculates the contact force generated by the interaction between the soft tissue O and the surgical instrument T. For example, the contact force is calculated by simulating the contact between the soft tissue O and the surgical instrument T after deformation. High-accuracy contact force calculation based on the high-accuracy deformation calculation result of the soft tissue O is possible. Note that in addition to the contact force, the pressure (gripping force) applied to the opening and closing / clamping part at the distal end of the surgical instrument T and the like can also be calculated. Within the scope of no contradiction, the pressure explained above can also be understood to be included in the contact force.
[0077] By using the high computing power of the processing unit 21 of the information processing device 2, the calculation cycle (arithmetic operation cycle) of the simulation can be reduced, in other words, the frame frequency of the simulation can be increased. Specifically, the processing unit 21 calculates the deformation of the model, the presence or absence of a collision, the contact force, the gravity, etc., with a cycle that is shorter than the calculation cycle assuming that the information processing device 1 has performed its calculations, and is the same as or longer than the control cycle of the tactile device 6. For example, the processing unit 21 calculates the simulation model with a predetermined cycle that is higher than conventional drawing processing and is equivalent to the motion control of a surgical robot. The example of the cycle described above is a cycle that is equal to or higher than 60 Hz and is approximately the same as 1 kHz (for example, substantially equal to 1 kHz).
[0078] The processing unit 11 of the information processing device 1 controls the tactile device 6 so that the contact force calculated by the processing unit 21 of the information processing device 2 is fed back to the user U (tactile control processing). The control cycle of the tactile device 6 may be the same as the cycle in which the processing unit 21 of the information processing device 2 calculates the simulation model. For example, by controlling the tactile device 6 at a fast cycle (short cycle) of 1 kHz or more, more accurate tactile feedback can be performed compared to when the tactile device 6 is controlled at a slow cycle (long cycle) of about 60 Hz. Tactile feedback corresponding to the droop of an organ when gravity is taken into account, etc. is also possible.
[0079] The processing unit 11 of the information processing device 1 generates a video (rendering process) including the soft tissue O simulated by the processing unit 21 of the information processing device 2. An example of a cycle of the rendering process is 60 Hz, etc. The rendering process may include a CG rendering process. Ray tracing may be used. A realistic video may be generated. The processing unit 11 may generate a video so that information about the contact force calculated by the processing unit 21 of the information processing device 2 is included in the video. Figure 4 This is explained.
[0080] Figure 4is a diagram showing an example of a video. Soft tissue O and surgical instruments T are displayed in the left portion of the monitor 7. Three surgical instruments T are shown as an example. Letters A, B, and C for distinguishing the surgical instruments T are displayed in association with each other. In the right portion of the monitor 7, information related to the contact force corresponding to the surgical instrument T, specifically, the value of the contact force (in N) is displayed as a numerical value and a bar.
[0081] Figure 5 is a diagram showing an example of a simulation cycle. The processing performed during one frame is schematically shown. In this example, the XPBD method described above is used for the simulation of soft tissue O.
[0082] In step S1, the simulation model is updated. For example, the soft tissue O is cut open, the mesh topology is changed, and the model of the soft tissue O is updated.
[0083] In step S2, simulation using the soft tissue O etc. updated in the previous step S1 is performed. The processing in step S2 is subdivided into a fixed number of sub-steps. Figure 5 In the example shown, the processing in sub-step S21 to sub-step S24 is repeatedly performed.
[0084] In substep S21, the position of the mesh part is predicted. In substep S22, a collision corresponding to the predicted position is detected. In substep S23, a so-called constraint projection is performed, the FEM-based constraints are solved based on XPBD, and the new position is calculated in parallel (substep S23a). In this example, an iterative process by the Jacobi method is performed. The calculation result is obtained as a collision response (substep S23b).
[0085] In sub-step S24, the velocity is calculated and updated based on the collision response (ie, the new position). After the processing of sub-step S24, the processing in sub-step S21 is performed again. This sub-step loop is repeated within the time step range of step S2.
[0086] In step S3, the simulation result in the previous step S2 is obtained. In this process, the simulation result includes the soft tissue O after deformation.
[0087] In step S4, the visual model is updated and reflected in the video display.
[0088] For example, the above-described processing is repeatedly performed at a fast cycle of 1 kHz or higher.
[0089] Return to reference Figure 2, necessary analog data is sent and received between information processing device 1 and information processing device 2. The analog data is sent and received via the all-optical communication network N. By using the all-optical communication network N, the delay in sending and receiving the analog data can be further suppressed (the delay can be further reduced) compared to when the all-optical communication network N is not used.
[0090] With the simulation system 100, by using the high computing power of the processing unit 21 of the information processing device 2, calculations of large simulation models can be performed, which are difficult to achieve only by the processing unit 11 of the information processing device 1. Calculations of models of the entire abdominal cavity, etc. are also possible. The deformation of soft tissue O, etc. can be accurately calculated. Due to the high computing power, a fine mesh model can be used. Therefore, the deformation of small soft tissue O, etc. can be accurately calculated. A simulation of a real surgical environment OE can be provided.
[0091] By using an all-optical communication network N, simulation data can be sent and received with low latency. It is also possible to provide simulations with real-time characteristics.
[0092] In one embodiment, the information processing device 1 and the information processing device 2 can use remote DMA to send and receive analog data to each other. In this case, the updated data is sent and received between the storage unit 12 of the information processing device 1 and the storage unit 22 of the information processing device 2 by remote DMA without CPU intervention, and the updated data is synchronized. By using remote DMA, the delay can be further reduced. Figure 6 This is explained.
[0093] Figure 6 2 is a diagram schematically showing remote DMA. In this example, in the information processing device 2 , the functions of the processing unit 21 and the storage unit 22 are implemented on the virtual layer 27 .
[0094] The simulation data (at least a part thereof) stored in the storage unit 12 of the information processing device 1 and the simulation data (at least a part thereof) stored in the storage unit 22 of the information processing device 2 can be synchronized through the path of the remote DMA indicated by the dotted arrow. For example, the motion data of the surgical instrument T processed by the processing unit 11 of the information processing device 1 and the data of the simulation model processed by the processing unit 21 of the information processing device 2 are shared with low latency. The effect of accelerating the simulation cycle can be further enhanced.
[0095] Figure 7: is a flowchart showing an example of processing (information processing method) performed in the simulation system. The processing in steps S31, S32, and S37 to S40 is performed in the edge region EF. The processing in steps S34 and S35 is performed in the cloud region CF. The processing in steps S33 and S36 is remote DMA.
[0096] In step S31 , surgical instrument operation data is acquired. The user U operates the haptic device 6 . The haptic device 6 acquires surgical instrument operation data corresponding to the user operation. Position / force information corresponding to the surgical instrument operation data is sent from the haptic device 6 to the processing unit 11 of the information processing apparatus 1 .
[0097] In step S32, the surgical instrument movement information is updated. The processing unit 11 of the information processing apparatus 1 updates the movement of the surgical instrument T in the surgical environment OE.
[0098] In step S33 , remote DMA is performed. The analog data in the storage unit 12 of the information processing device 1 is transferred to the storage unit 22 of the information processing device 2 .
[0099] In step S34, the simulation model is calculated. The processing unit 21 of the information processing device 2 calculates the deformation of the soft tissue O and calculates the contact force.
[0100] In step S35, the contact force and the model shape are updated. The processing unit 21 of the information processing device 2 updates the simulation model such as the contact force and the shape of the soft tissue O based on the calculation result of the simulation model.
[0101] In step S36, remote DMA is performed. The analog data in the storage unit 22 of the information processing device 2 is transferred to the storage unit 12 of the information processing device 1. After the process of step S36, the process proceeds to each of step S37 and step S39.
[0102] In step S37 , the haptic device 6 is controlled. The processing unit 11 of the information processing apparatus 1 controls the haptic device 6 so that the updated contact force is fed back to the user U.
[0103] In step S38 , the contact force is fed back. The contact force is fed back to the user U through the haptic device 6 (the force sense is presented).
[0104] In step S39, a video is generated. The processing unit 11 of the information processing device 1 generates a video including the updated soft tissue O and the like.
[0105] In step S40, the video is displayed. The monitor 7 displays the video.
[0106] When the processing in step S38 and step S40 is completed, the processing returns to step S31. A series of processing is repeatedly performed and the simulation is advanced.
[0107] 2. Modification
[0108] The disclosed technology is not limited to the above-described embodiments. Some modifications are described. For example, the simulation system 100 may include a plurality of information processing devices 1 used by different users U, respectively. Multi-base connection via the all-optical communication network N is possible. Figure 8 This is explained.
[0109] Figure 8 is a diagram showing a modified schematic configuration. The information processing device 1-1 and the information processing device 1-2 are exemplified as a plurality of information processing devices 1. The information processing device 1-1 and the information processing device 1-2 may be disposed separately from each other or may be disposed close to each other.
[0110] The user U who operates the information processing device 1-1 (the tactile device 6) is referred to as the user U-1 and is shown. The user U who operates the information processing device 1-2 (the tactile device 6) is referred to as the user U-2 and is shown. For example, the user U-1 is a surgeon and the user U-2 is an assistant. The operations of both the user U-1 and the user U-2 are reflected in the simulation. For example, the user U-1 and the user U-2 operate their respective surgical instruments to treat the same soft tissue O. Reference Fig. 9 This is explained.
[0111] Fig. 9 is a diagram schematically showing the flow of simulation. The surgical instrument T operated by the user U-1 is referred to as surgical instrument T-1 and is shown. The surgical instrument T operated by the user U-2 is referred to as surgical instrument T-2 and is shown.
[0112] In step S51, the simulation model is calculated. The processing unit 21 of the information processing device 2 calculates the simulation model based on the movement of the surgical instrument T-1 and the movement of the surgical instrument T-2. In step S52, the simulation model is updated. The updated data is transmitted to the information processing device 1-1 and the information processing device 1-2.
[0113] In step S53 , rendering and haptic control by the information processing apparatus 1 - 1 are performed. The processing unit 11 of the information processing apparatus 1 - 1 generates a video based on the updated simulation model, and controls the haptic device 6 .
[0114] In step S54 , rendering and haptic control by the information processing apparatus 1 - 2 are performed. The processing unit 11 of the information processing apparatus 1 - 1 generates a video based on the updated simulation model, and controls the haptic device 6 .
[0115] In step S55, the operation performed by the user U-1 is performed. The processing unit 11 of the information processing device 1-1 simulates the movement of the surgical instrument T-1 according to the operation of the tactile device 6 by the user U-1. As exemplified by the arrow extending from the dotted line, the movement of the surgical instrument T-1 changes. The movement data of the surgical instrument T-1 is transmitted to the information processing device 2.
[0116] In step S56, the operation performed by the user U-2 is performed. The processing unit 11 of the information processing device 1-2 simulates the movement of the surgical instrument T-2 according to the operation of the tactile device 6 by the user U-2. As exemplified by the arrow extending from the dotted line, the movement of the surgical instrument T-2 is changed. The movement data of the surgical instrument T-2 is transmitted to the information processing device 2.
[0117] In step S57, the motion data of the surgical instrument T on the information processing apparatus 2 side is updated.
[0118] The various processes are repeatedly executed with the processes from step S51 to step S57 as one loop. Fig. 9 The processing in steps S58 to S61 shown is similar to the processing in steps S52 to S54, so the description will not be repeated.
[0119] For example, as explained above, multiple users U may use different surgical instruments T to treat the same soft tissue O. Surgical training may be performed by multiple users U.
[0120] The simulation described above can be applied to surgical robot simulation. In the surgical robot simulation, the user U operates the robot arm supporting the surgical instrument T via the haptic device 6. Since the contact force is fed back to the user U, it is also possible to evaluate, for example, whether the robot arm and the organ are operated with appropriate operating force and whether excessive force is not applied, thereby learning safe operation of the robot.
[0121] In one embodiment, simulation can be incorporated into the master-slave system. Fig.10 This embodiment will be described.
[0122] Fig.10 2 is a diagram showing a schematic configuration of a modification. A simulation system 100 includes a leader device 3 , a follower device 4 , and an information processing device 2 .
[0123] The basic components of the leader device 3 are the same as those of the information processing device 1 described above. The communication control unit 14 of the leader device 3 also controls the communication between the leader device 3 and the follower device 4 .
[0124] The follower device 4 includes a camera 8, a robot arm 9, a processing unit 41, an IF unit 43, and a communication control unit 44. The camera 8 captures an image of a surgical field, and the robot arm 9 supports a surgical instrument T.
[0125] The processing unit 41 acts as an overall control unit for controlling the elements of the follower device 4. The processing unit 41 observes a slave environment including the patient's status based on the video of the camera 8. The slave environment may also include non-visual information, such as the patient's blood pressure and heart rate. The processing unit 41 controls the robotic arm 9 and switches the operation of the position of the camera 8 according to the operation of the tactile device 6 in the leader device 3.
[0126] The IF unit 43 provides an interface between the processing unit 41 and the camera device 8 and the robot arm 9 .
[0127] The communication control unit 44 controls the communication between the follower device 4 and the leader device 3. Note that the communication between the follower device 4 and the information processing device 2 may also be controlled.
[0128] The leader device 3 and the follower device 4 serve as a master device and a slave device capable of performing bidirectional control. The surgical operation is advanced by controlling the robotic arm 9 supporting the surgical instrument T using a master-slave scheme. The video of the camera 8 of the follower device 4 is sent from the follower device 4 to the leader device 3 and displayed on the monitor 7 of the leader device 3. Information about other slave environments is also sent from the follower device 4 to the leader device 3 and presented. The position / force information from the tactile device 6 of the leader device 3 is sent from the leader device 3 to the follower device 4 and is reflected in the control of the robotic arm 9. The position / force information from the robotic arm 9 is sent from the follower device 4 to the leader device 3 and fed back to the user via the tactile device 6 of the leader device 3.
[0129] In the information processing device 2, the high-accuracy simulation model described above is calculated and updated. Fig.10 In the simulation system 100 shown, the results of the simulation are reflected in the control of the robot arm 9. For example, when the position / force information of the haptic device 6 from the leader device 3 is sent to the follower device 4, the control of the robot arm 9 is limited to ensure safety.
[0130] Specifically, the processing unit 21 of the information processing device 2 performs safety determination processing based on the calculation result of the simulation model. For example, the processing unit 21 determines whether the operation of the robot arm 9 of the follower device 4 performed via the leader device 3 is dangerous. When it is determined that the operation is dangerous, the processing unit 21 intervenes in the operation of the follower device 4 by the leader device 3. For example, the processing unit 21 corrects the position / force information sent from the leader device 3 to the follower device 4 so that the movement amount of the robot arm 9 is limited and the operation force is limited. The corrected position / force information is sent from the leader device 3 to the follower device 4. The determination result and control result of the processing unit 21 of the information processing device 2 are notified to the leader device 3 and the follower device 4. Note that in the safety monitoring described above, while predicting the future movement based on the motion data of the tactile device 6, a state slightly in the future than the current time can be simulated. By using the high computing power of the information processing device 2, the simulation described above can be performed in real time. Safety can be further improved.
[0131] 3. Hardware Configuration Example
[0132] Fig.11 1 is a diagram showing an example of the hardware configuration of the device. The information processing device 1, the information processing device 2, etc. described above are composed of, for example, Fig.11 The computer 1000 shown is implemented.
[0133] The computer 1000 includes a CPU / GPU 1100 , a RAM 1200 , a ROM (Read Only Memory) 1300 , an HDD (Hard Disk Drive) 1400 , a communication interface 1500 , and an input / output interface 1600 . The units of the computer 1000 are connected by a bus 1050 .
[0134] The CPU / GPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls the units. For example, the CPU / GPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing corresponding to the various programs.
[0135] The ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU / GPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000 , and the like.
[0136] The HDD 1400 is a computer-readable recording medium that non-transitorily records a program to be executed by the CPU / GPU 1100 , data used by the program, etc. Specifically, the HDD 1400 is a recording medium that records a program for the information processing method according to the present disclosure, which is an example of the program data 1450 .
[0137] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU / GPU 1100 receives data from other devices via the communication interface 1500 or transmits data generated by the CPU / GPU 1100 to other devices.
[0138] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU / GPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU / GPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. When the device is an information processing device 1, the CPU / GPU 1100 receives motion data from the tactile device 6 via the input / output interface 1600 or sends a simulated contact force to the tactile device. The input / output interface 1600 can be used as a medium interface for reading a program recorded in a computer-readable predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a DVD (digital versatile disc) or a PD (phase change rewritable disc), a magneto-optical recording medium such as an MO (magneto-optical disc), a magnetic tape medium, a magnetic recording medium, or a semiconductor memory.
[0139] When the computer 1000 is used as the information processing device 1 and the information processing device 2 described above, the CPU / GPU 1100 of the computer 1000 executes the program loaded on the RAM 1200 to realize the functions of the processing unit 11 and the processing unit 21. The program may be stored in the HDD 1400. Note that the CPU / GPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, the program may be acquired from another device via the external network 1550.
[0140] The components explained above may be configured using general-purpose members, or may be configured by hardware dedicated to the functions of the components. The configurations explained above may be appropriately changed according to the technical level each time the configurations are implemented.
[0141] 4. Effect examples
[0142] For example, as described below, the techniques described above are specified. One of the disclosed techniques is a simulation system 100. As described with reference to Figure 1 , Figure 2 As described in FIG. 1 , the simulation system 100 includes an information processing device 1 (first information processing device) and an information processing device 2 (second information processing device) that provide a simulation of a surgical operation in cooperation with each other. The information processing device 1 and the information processing device 2 transmit and receive simulation data to and from each other via an all-optical communication network N.
[0143] With the simulation system 100 described above, simulation is provided by the cooperation of the information processing device 1 and the information processing device 2. By using the all-optical communication network N, the delay of the communication between the information processing device 1 and the information processing device 2 can be suppressed (can be reduced). Therefore, for example, the simulation performance can be further improved compared to when the simulation is provided only by the information processing device 1. In particular, the simulation system 100 is useful for the following simulation: the simulation has a function such as tactile presentation that requires real-time characteristics, and the simulation system 100 can perform simulations such as deformation of flexible objects or delicate tissues with higher accuracy, for example.
[0144] As reference Figure 6 As described in , the information processing device 1 and the information processing device 2 can transmit and receive analog data to and from each other via remote DMA. This makes it possible to further reduce delays.
[0145] As reference Figure 1 , Figure 2 , Figure 5 As described in, for example, the information processing device 2 can calculate a simulation model including a soft tissue O (e.g., an organ). The calculation of the simulation model can include the calculation of the deformation of the soft tissue O. The information processing device 2 can calculate the contact force generated by the interaction between the soft tissue O and the surgical instrument T. For example, the simulation performance can be improved by performing the processing with a large calculation load described above by the information processing device 2. It is also possible to perform a fine simulation at a cycle that is higher in speed than the cycle used in general video processing and is suitable for the calculation of force sense, such as 1 kHz or higher.
[0146] As reference Figures 1 to 3As described in, etc., the information processing device 1 can control the tactile device 6 operated by the user U so that the contact force calculated by the information processing device 2 is fed back to the user U. The information processing device 2 can calculate the contact force at a predetermined period (further calculating the deformation of the model, the presence or absence of a collision, gravity, etc.). The information processing device 1 can control the tactile device 6 so that the contact force calculated by the information processing device 2 is fed back to the user U at this period. The predetermined period may be shorter than the calculation period in the following case: it is assumed that the information processing device 1 has performed calculations on the simulation model including the calculation of the contact force, and the predetermined period may be the same as or longer than the control period of the tactile device 6. The predetermined period may be a period of 60 Hz or higher and approximately the same as 1 kHz. Accurate tactile feedback can be performed.
[0147] The haptic device 6 may include a joint 61 in which at least one of acceleration and force is controlled. The haptic device 6 may include a joint 61 provided with at least one of a six-axis force sensor and a torque sensor. By using the haptic device 6 described above, more accurate haptic feedback may be performed.
[0148] As reference Figure 8 As described in FIG. 1 and FIG. 2 , the information processing device 1 may be a plurality of information processing devices 1 (eg, information processing device 1-1 and information processing device 1-2) used by different users U (eg, user U-1 and user U-2). A plurality of users U may perform a simulation of a surgical operation.
[0149] As reference Fig.10 As described in , the surgical operation may be a surgical operation for controlling the robot arm 9 supporting the surgical instrument T through a master-slave scheme, and the result of the simulation may be reflected in the control of the robot arm 9. For example, as described above, the simulation may be incorporated in the master-slave system and used.
[0150] Reference Figure 1 , Figure 2 The information processing device 2 described in the above is also one of the disclosed technologies. The information processing device 2 provides a simulation of a surgical operation in cooperation with the information processing device 1 (another information processing device). The information processing device 2 includes: a processing unit 21 that calculates a simulation model including soft tissue O; and an optical transmission control unit that controls the sending of simulation data to the information processing device 1 via the all-optical communication network N and the receiving of simulation data from the information processing device 1. Such an information processing device 2 can also improve the simulation performance, as described above.
[0151] Reference Figure 7The information processing method described in the above is also one of the disclosed technologies. In the information processing method, information processing device 1 (first information processing device) and information processing device 2 (second information processing device) cooperate with each other to provide a simulation of a surgical operation. The information processing method includes information processing device 1 and information processing device 2 sending and receiving simulation data to each other via an all-optical communication network N (step S33 and step S36). Using the information processing method described above, the simulation performance can also be improved, as described above.
[0152] Note that the effects described in the present disclosure are merely examples and are not limited to the disclosed contents. There may be other effects.
[0153] Although the embodiments of the present disclosure are described above, the technical scope of the present disclosure is not limited to the embodiments described above. Various changes are possible without departing from the gist of the present disclosure. The components in different embodiments and modifications can be appropriately combined.
[0154] Note that the present technology can also adopt the following configurations.
[0155] (1) A simulation system comprising:
[0156] A first information processing device and a second information processing device that provide a simulation of a surgical operation in cooperation with each other, wherein
[0157] The first information processing device and the second information processing device transmit and receive analog data to and from each other via an all-optical communication network.
[0158] (2) The simulation system according to (1), wherein:
[0159] The first information processing device and the second information processing device transmit and receive the simulation data to and from each other through remote DMA (Direct Memory Access).
[0160] (3) The simulation system according to (1) or (2), wherein:
[0161] The second information processing device calculates a simulation model including soft tissue.
[0162] (4) The simulation system according to (3), wherein:
[0163] The calculation of the simulation model includes the calculation of the deformation of the soft tissue.
[0164] (5) The simulation system according to (4), wherein:
[0165] The soft tissue includes organs.
[0166] (6) The simulation system according to (4) or (5), wherein:
[0167] The second information processing device calculates a contact force generated by an interaction between the soft tissue and a surgical instrument.
[0168] (7) The simulation system according to (6), wherein:
[0169] The first information processing apparatus controls a haptic device operated by a user so that the contact force calculated by the second information processing apparatus is fed back to the user.
[0170] (8) The simulation system according to (7), wherein:
[0171] the second information processing device calculates the contact force at a predetermined period,
[0172] The first information processing device controls the haptic device so that the contact force calculated by the second information processing device is fed back to the user in the period, and
[0173] The predetermined cycle is the same as or longer than a control cycle of the haptic device and is shorter than a calculation cycle assuming that the first information processing apparatus has performed calculation of the simulation model including calculation of the contact force.
[0174] (9) The simulation system according to (8), wherein:
[0175] The predetermined period is a period that is equal to or higher than 60 Hz and is substantially the same as 1 kHz.
[0176] (10) The simulation system according to (8) or (9), wherein:
[0177] The haptic device includes a joint at which at least one of acceleration and force is controlled.
[0178] (11) The simulation system according to any one of (8) to (10), wherein
[0179] The haptic device includes a joint provided with at least one of a six-axis force sensor and a torque sensor.
[0180] (12) The simulation system according to any one of (1) to (11), wherein:
[0181] The first information processing devices are a plurality of first information processing devices used by different users.
[0182] (13) The simulation system according to any one of (1) to (6), wherein:
[0183] The surgical operation is a surgical operation in which a robotic arm supporting a surgical instrument is controlled using a master-slave scheme, and
[0184] The results of the simulation are reflected in the control of the robot arm.
[0185] (14) An information processing device that provides a simulation of a surgical procedure in cooperation with another information processing device, the information processing device comprising:
[0186] a processing unit that calculates a simulation model including soft tissue; and
[0187] An optical transmission control unit controls sending and receiving analog data to and from the other information processing device via the all-optical communication network.
[0188] (15) An information processing method for providing a simulation of a surgical operation by a first information processing device and a second information processing device cooperating with each other, the information processing method comprising:
[0189] The first information processing device and the second information processing device transmit and receive analog data to and from each other via the all-optical communication network.
[0190] List of Figure Reference Symbols
[0191] 1 Information processing equipment
[0192] 11 Processing Unit
[0193] 12 Storage Units
[0194] 13 IF Unit
[0195] 14 Communication control unit
[0196] 15 Optical transmission control unit
[0197] 16 Optical transmission device
[0198] 2 Information processing equipment
[0199] 21 Processing Units
[0200] 22 Storage Units
[0201] 24 Communication control unit
[0202] 25 Optical transmission control unit
[0203] 26 Optical transmission device
[0204] 27 Virtual Layer
[0205] 3 Leader device (master device)
[0206] 4 Follower devices (slave devices)
[0207] 41 Processing Units
[0208] 43 IF Unit
[0209] 44 Communication control unit
[0210] 6. Tactile Devices
[0211] 61 Joints
[0212] 611 Motor
[0213] 612 Encoder
[0214] 613 Sensor
[0215] 614 Motor Control Unit
[0216] 615 Driver
[0217] 62 Haptic Control Unit
[0218] 7 Monitor
[0219] 8 Camera Device
[0220] 9 Robotic Arm
[0221] 100 Analog System
[0222] 1000 computers
[0223] 1050 bus
[0224] 1100CPU / GPU
[0225] 1200 RAM
[0226] 1300ROM
[0227] 1400HDD
[0228] 1450 Program Data
[0229] 1500 communication interface
[0230] 1600 input / output interface
[0231] 1650 Input and output devices
[0232] C Camera Device
[0233] F Field of view
[0234] N All-optical communication network
[0235] O Soft tissue
[0236] R Robotic Arm
[0237] T Surgical Instruments
[0238] U User
Claims
1. A simulation system, comprising: A first information processing device and a second information processing device that provide a simulation of a surgical operation in cooperation with each other, wherein The first information processing device and the second information processing device transmit and receive analog data to and from each other via an all-optical communication network.
2. The simulation system according to claim 1, wherein: The first information processing device and the second information processing device transmit and receive the simulation data to and from each other through remote DMA (Direct Memory Access).
3. The simulation system according to claim 1, wherein: The second information processing device calculates a simulation model including soft tissue.
4. The simulation system according to claim 3, wherein: The calculation of the simulation model includes the calculation of the deformation of the soft tissue.
5. The simulation system according to claim 4, wherein: The soft tissue includes organs.
6. The simulation system according to claim 4, wherein: The second information processing device calculates a contact force generated by an interaction between the soft tissue and a surgical instrument.
7. The simulation system according to claim 6, wherein: The first information processing apparatus controls a haptic device operated by a user so that the contact force calculated by the second information processing apparatus is fed back to the user.
8. The simulation system according to claim 7, wherein: the second information processing device calculates the contact force at a predetermined period, The first information processing device controls the haptic device so that the contact force calculated by the second information processing device is fed back to the user in the period, and The predetermined cycle is the same as or longer than a control cycle of the haptic device and is shorter than a calculation cycle assuming that the first information processing apparatus has performed calculation of the simulation model including calculation of the contact force.
9. The simulation system according to claim 8, wherein: The predetermined period is a period that is equal to or higher than 60 Hz and is substantially the same as 1 kHz.
10. The simulation system according to claim 7, wherein: The haptic device includes a joint at which at least one of acceleration and force is controlled.
11. The simulation system according to claim 7, wherein: The haptic device includes a joint provided with at least one of a six-axis force sensor and a torque sensor.
12. The simulation system according to claim 1, wherein: The first information processing devices are a plurality of first information processing devices used by different users.
13. The simulation system according to claim 1, wherein: The surgical operation is a surgical operation in which a robotic arm supporting a surgical instrument is controlled using a master-slave scheme, and The results of the simulation are reflected in the control of the robot arm.
14. An information processing device that provides a simulation of a surgical procedure in cooperation with another information processing device, the information processing device comprising: a processing unit that calculates a simulation model including soft tissue; as well as An optical transmission control unit controls sending and receiving analog data to and from the other information processing device via the all-optical communication network.
15. An information processing method for providing a simulation of a surgical operation by a first information processing device and a second information processing device cooperating with each other, the information processing method comprising: The first information processing device and the second information processing device transmit and receive analog data to and from each other via the all-optical communication network.
