Information processing device, information processing method, and program
The information processing device addresses the limitations of current training methods by associating real and virtual space position information and providing feedback, enabling a more realistic and flexible training experience for medical tool operation.
Patent Information
- Application Number
- PCT/JP2024/014126
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-22
- Filing Date
- 2024-04-05
- Publication Date
- 2024-11-28
AI Technical Summary
Current training methods for medical tools like endoscopes face limitations in replicating the texture and biological events of actual organs, and require dedicated training tools, restricting the selection of medical tools and realism in simulations.
An information processing device that associates three-dimensional position information in real space with virtual space, tracking the medical tool's position, and providing feedback through a feedback control unit to simulate realistic interactions and biological events, allowing users to freely select medical tools and enhancing training realism.
This solution eliminates the need for dedicated training tools, provides a more realistic training experience by accurately simulating the interaction and texture of actual biological organs, and allows users to freely choose medical tools for training.
Smart Images

Figure JP2024014126_28112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and program
[0001] The present disclosure relates to an information processing device, an information processing method, and a program.
[0002] In recent years, surgeries and examinations using medical tools such as endoscopes have been performed. Furthermore, technologies to support training in the operation of medical tools such as endoscopes have also been developed. For example, Patent Literature 1 discloses a portable surgical training device that supports training in the operation of an endoscope.
[0003] Japanese Patent Application Laid-Open No. 2022-140461
[0004] However, as disclosed in Patent Document 1, when training is performed using a physical model organ, there is a limit to the reproduction of the texture and the like of an actual biological organ.
[0005] According to one aspect of the present disclosure, there is provided an information processing device comprising: a model processing unit that performs scaling processing to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
[0006] According to another aspect of the present disclosure, there is provided an information processing method including: a processor performing a scaling process to associate three-dimensional position information in a real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; tracking the three-dimensional position of a medical tool operated by a user in the real space; and controlling the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
[0007] According to another aspect of the present disclosure, there is provided a program that causes a computer to function as an information processing device, comprising: a model processing unit that performs scaling processing to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
[0008] FIG. 1 is a diagram for describing an overview of an information processing system 1 according to an embodiment of the present disclosure. FIG. 1 is a block diagram for describing an example functional configuration of an information processing device 10 according to the embodiment. FIG. 1 is a diagram for describing scaling processing according to the embodiment. FIG. 1 is a diagram for describing scaling processing according to the embodiment. FIG. 1 is a diagram for describing tracking processing according to the embodiment. FIG. 1 is a diagram for describing an example configuration of a feedback device 40 according to the embodiment. FIG. 1 is a diagram for describing operational restrictions of an endoscope 20 according to the embodiment. FIG. 1 is a diagram for describing operation of a rotation feedback structure 420 according to the embodiment. FIG. 1 is a flowchart for describing codebook learning according to the embodiment. FIG. 1 is a diagram for describing a flow of codebook learning according to the embodiment. FIG. 1 is a diagram for describing processing processing according to the embodiment. FIG. 1 is a flowchart for describing a flow of loading a three-dimensional model 60 according to the embodiment. FIG. 1 is a flowchart for describing a flow of starting and ending tracking processing according to the embodiment. FIG. 1 is a flowchart for tracking processing and feedback control according to the embodiment. FIG. 1 is a flowchart for describing a flow of feedback by the feedback device 40 according to the embodiment. FIG. 2 is a block diagram for describing an example hardware configuration of an information processing device 90 according to the embodiment.
[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.
[0010] In addition, in this specification and drawings, when multiple identical components are to be described separately, letters or the like may be added to the end of the reference numerals. On the other hand, when it is not necessary to distinguish between multiple identical components, the letters or the like may be omitted and a description common to all of the multiple identical components may be given.
[0011] The description will be given in the following order: 1. Embodiment 1.1. Overview 1.2. Example of functional configuration of information processing device 10 1.3. Details of functions 2. Example of hardware configuration 3. Summary
[0012] <1. Embodiment> <<1.1. Overview>>
[0013] As described above, in recent years, techniques have been developed to support training in the operation of medical tools such as endoscopes.
[0014] However, as disclosed in Patent Document 1, when a physical model is used, the texture of the material (plastic, wood, etc.) that forms the model is strong, making it difficult to realistically reproduce the texture of an actual biological organ.
[0015] Furthermore, when using a physical model, it is difficult to reproduce biological events such as the movement of biological organs and bleeding.
[0016] Furthermore, when using a physical model, it is necessary to prepare as many models as there are biological organs to be trained.
[0017] On the other hand, computer-based simulation-based training support technology has also been developed.
[0018] However, most simulation-based training support technologies use computer graphics (CG) like games, which are poor at reproducing the texture of actual biological organs.
[0019] Furthermore, in order to move the endoscope in the virtual space as intended by the user, it is necessary to sense the movement of the endoscope operated by the user.
[0020] The sensing method generally involves using a sensor to acquire the position and pose of the tip of the endoscope, but it is difficult to attach such a sensor to an endoscope that will actually be used in surgery, etc.
[0021] For this reason, simulation-based training support technology uses endoscopes equipped with sensors specifically for training purposes, meaning that trainees cannot freely select the endoscopes they use for practice.
[0022] The technical concept of one embodiment of the present disclosure was conceived with the above points in mind, and eliminates limitations on medical tools used in training and provides a more realistic training experience.
[0023] First, an overview of an information processing system 1 according to an embodiment of the present disclosure will be described with reference to FIG.
[0024] As shown in FIG. 1, an information processing system 1 according to this embodiment includes an information processing device 10, an endoscope 20, a chamber 30, a feedback device 40, and a display device 50.
[0025] (Information processing device 10) As will be described in detail later with reference to Figure 2, the information processing device 10 of this embodiment includes a model processing unit 110 that performs scaling processing to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model 60 of a biological organ in virtual space, a tracking unit 120 that tracks the three-dimensional position of a medical tool operated by a user in real space, and a feedback control unit 130 that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model 60 of the biological organ based on the three-dimensional position of the tracked medical tool.
[0026] (Endoscope 20) The endoscope 20 according to this embodiment is an example of a medical tool.
[0027] The endoscope 20 according to this embodiment includes a tip portion 210 to which an imaging device is attached, a controller 220 used to operate the tip portion 210 and the like, and a cable portion 230 connecting the tip portion 210 and the controller 220.
[0028] An endoscope 20 operated by a user 80 is inserted into the chamber 30 via a feedback device 40 .
[0029] (Chamber 30) The chamber 30 according to this embodiment is, for example, a box-shaped housing used to track the position and pose of the endoscope 20 in real space.
[0030] The model processing unit 110 according to this embodiment performs scaling processing to associate the three-dimensional position of the chamber 30 placed in real space with the three-dimensional position information of the three-dimensional model 60 relating to the biological organ.
[0031] Furthermore, the tracking unit 120 according to this embodiment tracks the three-dimensional position of the tip portion 210 of the endoscope 20 within the chamber 30 .
[0032] Furthermore, the tracking unit 120 in this embodiment estimates the three-dimensional position of the tip 210 of the endoscope 20 in the virtual space based on the three-dimensional position of the tip 210 of the endoscope 20 within the chamber 30 and the results of the scaling process.
[0033] According to this process, an endoscope dedicated to training as described above is not required, and the user 80 can freely select the endoscope 20 according to his / her purpose and preference.
[0034] (Feedback Device 40) The feedback device 40 according to this embodiment provides feedback regarding the interaction between the endoscope 20 and the three-dimensional model 60 of a biological organ, based on the control of the feedback control unit 130.
[0035] The feedback according to this embodiment will be described in detail later.
[0036] (Display device 50) The display device 50 according to this embodiment displays an image of a three-dimensional model 60 of a biological organ according to the position and posture of the endoscope 20, based on the control by the interface control unit 140 (see FIG. 2) provided in the information processing device 10.
[0037] The outline of the information processing system 1 according to this embodiment has been described above.
[0038] According to the information processing system 1 of this embodiment, the user 80 can obtain an image of a three-dimensional model 60 of a biological organ according to the position and posture of the endoscope 20 within the chamber 30 by operating the endoscope 20.
[0039] Furthermore, texture conversion, which will be described later, can provide images that realistically reproduce actual biological organs, making it possible to provide the user 80 with a realistic training experience.
[0040] <<1.2. Example of Functional Configuration of Information Processing Apparatus 10>> Next, an example of the functional configuration of the information processing apparatus 10 according to this embodiment will be described.
[0041] FIG. 2 is a block diagram illustrating an example of the functional configuration of the information processing device 10 according to this embodiment.
[0042] As shown in FIG. 2, the information processing device 10 according to this embodiment includes a control unit 100 .
[0043] The control unit 100 according to this embodiment is realized by various processors, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a video processor, etc. The control unit 100 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
[0044] As shown in FIG. 2, the control unit 100 according to this embodiment also functions as a model processing unit 110, a tracking unit 120, a feedback control unit 130, and an interface control unit 140.
[0045] (Model Processing Unit 110) The model processing unit 110 according to this embodiment performs scaling processing and the like to associate three-dimensional position information in real space with three-dimensional position information of the three-dimensional model 60 of a biological organ in virtual space.
[0046] (Tracking Unit 120) The tracking unit 120 according to this embodiment tracks the three-dimensional position of a medical tool (for example, the endoscope 20) operated by the user 80 in real space.
[0047] In addition, the tracking unit 120 according to this embodiment estimates the three-dimensional position of the medical tool in the virtual space based on the three-dimensional position of the medical tool in the real space and the result of the scaling process performed by the model processing unit 110.
[0048] (Feedback control unit 130) The feedback control unit 130 in this embodiment controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model 60 of the biological organ based on the three-dimensional position of the medical tool tracked by the tracking unit 120.
[0049] (Interface Control Unit 140) The interface control unit 140 according to this embodiment controls the presentation of images and the like related to the three-dimensional model 60 of a biological organ according to the three-dimensional position of the medical tool in the virtual space.
[0050] The interface control unit 140 also controls the reception of various operations by the user 80 or the like.
[0051] As shown in FIG. 2, the information processing device 10 according to this embodiment includes a model database 150 and a codebook database 160 .
[0052] (Model database 150) The model database 150 according to this embodiment is a database that stores three-dimensional position information of a three-dimensional model 60 relating to a biological organ, a texture image (CG of a setting photographed at a certain three-dimensional position) associated with the three-dimensional position information, and the like.
[0053] (Codebook Database 160) The codebook database 160 according to this embodiment is a database that stores a codebook latent 74, which will be described later.
[0054] The functional configuration of the information processing device 10 according to the present embodiment has been described above. Note that the functional configuration described above with reference to Fig. 2 is merely an example, and the functional configuration of the information processing device 10 according to the present embodiment is not limited to this example.
[0055] For example, the functions of the model processing unit 110, the tracking unit 120, the feedback control unit 130, and the interface control unit 140 may be distributed and implemented in multiple devices.
[0056] The functional configuration of the information processing device 10 according to this embodiment can be flexibly modified according to specifications and operations.
[0057] <<1.3. Detailed Functions>> Next, each function of the information processing device 10 according to this embodiment will be described in detail.
[0058] (Scaling Processing) First, the scaling processing by the model processing unit 110 will be described.
[0059] The tracking unit 120 according to this embodiment first estimates the three-dimensional position of the endoscope 20 in real space using SLAM (Simultaneous Localization and Mapping), and then estimates the three-dimensional position of the endoscope 20 in virtual space based on the results of this estimation.
[0060] As a pre-processing step, the model processing unit 110 performs scaling processing to align the size of the three-dimensional model 60 of the biological organ with the size of the chamber 30 .
[0061] 3 and 4 are diagrams for explaining the scaling process according to this embodiment.
[0062] As shown in FIG. 3, in a virtual space expressed by X^, Y^, and Z^, the axis on which the three-dimensional model 60 of the biological organ has the greatest length is denoted by L^, and the length of the diagonal is denoted by R^.
[0063] In addition, in a real space expressed by X, Y, and Z corresponding to X^, Y^, and Z^, respectively, the axis corresponding to the above L^ is defined as L, and the length of the diagonal is defined as R.
[0064] In this case, as shown in FIG. 4, the model processing unit 110 first scales R^ based on R (S101).
[0065] Next, the model processing unit 110 assumes that Z and L are complete (S102).
[0066] Furthermore, the model processing unit 110 assumes that Z^ and L^ are available (S103).
[0067] Next, the model processing unit 110 aligns Z^ and Z (S104).
[0068] Next, the model processing unit 110 defines an insertion point 65 (illustrated in FIG. 3) of the endoscope 20 in the three-dimensional model 60 of the biological organ (S105).
[0069] The insertion point 65 is used as an initial position when the tracking unit 120 estimates (tracks) the three-dimensional position of the endoscope 20 in the virtual space.
[0070] (Tracking Processing) Next, the processing of tracking the three-dimensional position of the endoscope 20 in real space / virtual space by the tracking unit 120 according to this embodiment will be described.
[0071] The tracking unit 120 starts the tracking process based on the detection of the entry of the endoscope 20 into the chamber 30. A method for detecting the entry of the endoscope 20 into the chamber 30 will be described later.
[0072] When the entry of the endoscope 20 into the chamber 30 is detected, the tracking unit 120 performs initialization to determine that the position of the endoscope 20 in the virtual space is at the insertion point described above.
[0073] The tracking process according to this embodiment uses a self-location estimation process using SLAM, but does not involve a mapping process.
[0074] 5 and 6 are diagrams for explaining the tracking process according to this embodiment.
[0075] As shown in FIG. 5, a plurality of unique tags 35 are arranged on the inner wall of a chamber 30 according to this embodiment, from each of which different feature quantities can be extracted.
[0076] The unique tag 35 may be, for example, a QR code (registered trademark), ArUco, or the like.
[0077] The unique tag 35 may be printed, projected, or displayed on a display.
[0078] The tracking unit 120 according to this embodiment estimates the three-dimensional position of the endoscope in real space (inside the chamber 30 ) based on the feature amount of the unique tag 35 .
[0079] More specifically, the tracking unit 120 in this embodiment may estimate the three-dimensional position of the endoscope 20 in real space based on changes in the features of the unique tag 35 between multiple images (frames) taken in time series by the endoscope 20.
[0080] In this example, it is assumed that the distal end portion 210 of the endoscope 20 moves from position <1> to position <2> shown in FIG.
[0081] In this case, as the movement is performed, multiple frames may be acquired by the endoscope 20. In this example, it is assumed that multiple frames, including frame F and F-(1 to 10), are captured at position <2>.
[0082] The tracking unit 120 extracts the feature amount of the unique tag 35 captured in each frame.
[0083] The tracking unit 120 compares the feature amounts extracted from the frame to be processed with the feature amounts extracted from the frames acquired immediately before the frame to be processed (for example, frames F and F-1).
[0084] Next, the tracking unit 120 calculates the relative position pose between frames (the amount of change in the position pose between frames) based on the above comparison, thereby estimating the position pose R of the endoscope 20 in real space at the time each frame is acquired.
[0085] Furthermore, the tracking unit 120 estimates the position pose S of the endoscope 20 in the virtual space based on the estimated position pose R of the endoscope 20 in the real space and the result of the scaling process.
[0086] Although an example of the tracking process has been described above, the tracking process according to this embodiment is not limited to this example.
[0087] For example, the unique tag 35 may be associated with an ID and absolute position information from the insertion opening of the endoscope 20 in the chamber 30. In this case, the tracking unit 120 can realize more accurate tracking processing based on the absolute position information.
[0088] Furthermore, after estimating the position pose S in the tracking process, the tracking unit 120 detects the interaction between the endoscope 20 and the three-dimensional model 60 of the biological organ in the virtual space.
[0089] An example of the interaction is a collision between the endoscope 20 and a three-dimensional model of a living organ.
[0090] When a collision is detected, the model processing unit 110 according to this embodiment performs a mesh deformation calculation (changing parameters of the mesh structure) on the three-dimensional model 60 of the biological organ.
[0091] That is, the information processing device 10 according to this embodiment can simulate in a virtual space how the shape of a biological organ changes when the endoscope 20 collides with the biological organ.
[0092] Furthermore, when a collision is detected, the model processing unit 110 determines whether or not a bio-event caused by the collision has occurred.
[0093] The biological event may be, for example, bleeding from a biological organ, fogging of a camera provided in the endoscope 20, or a malfunction of the endoscope.
[0094] The model processing unit 110 may determine whether or not a bio-event has occurred based on a predefined preset scenario of the training.
[0095] On the other hand, the model processing unit 110 may determine whether or not a bio-event has occurred based on the calculated deformation amount of the mesh.
[0096] For example, the model processing unit 110 may determine that a bio-event has occurred when the deformation amount of the mesh exceeds a threshold value.
[0097] (Feedback Processing) Next, the feedback processing according to this embodiment will be described.
[0098] The feedback control unit 130 according to this embodiment controls the presentation of feedback corresponding to a collision when a collision is detected.
[0099] The feedback corresponding to the collision includes a movement restriction of the medical tool in real space.
[0100] FIG. 7 is a diagram for explaining an example of the configuration of the feedback device 40 according to this embodiment.
[0101] The feedback device 40 includes at least a stopper 410 and an actuator 415 that changes the position of the stopper 410 .
[0102] FIG. 7 illustrates an example in which the feedback device 40 includes stoppers 410A and 410B and actuators 415A and 415B.
[0103] The feedback control unit 130 according to this embodiment controls the actuators 415A and 415B to restrict the operation of the endoscope 20 when a collision is detected.
[0104] FIG. 8 is a diagram for explaining the operation restrictions of the endoscope 20 according to this embodiment.
[0105] The upper part of FIG. 8 shows an example of the configuration of the feedback device 40 in a situation where no collision occurs.
[0106] As shown in the upper part of FIG. 8, in a situation where no collision occurs, the stoppers 410A and 410B are placed at positions where they do not interfere with the movement of the cable portion of the endoscope 20.
[0107] On the other hand, the lower part of FIG. 8 shows an example of the configuration of the feedback device 40 in a situation where a collision occurs.
[0108] When a collision occurs, the feedback control unit 130 sends a control signal to the feedback device 40 to extend the actuators 415A and 415B, as shown in the lower part of FIG.
[0109] This causes actuator 415A to move stopper 410A downward, and actuator 415B to move stopper 410B upward.
[0110] By the above-described control, the stoppers 410A and 410B pinch the cable portion 230 of the endoscope 20, thereby restricting the movement of the endoscope 20 so that it cannot be inserted further than the current position.
[0111] As described above, the feedback control unit 130 according to this embodiment can allow the user 80 to tactilely sense the occurrence of a collision by restricting the operation of the endoscope 20 .
[0112] The movement restriction of the endoscope 20 may be a restriction on the bending of the distal end portion 210 .
[0113] When a collision occurs, the feedback control unit 130 may perform control so that the three-dimensional position of the endoscope in the virtual space is not changed, mesh deformation is not performed, or the like, even if the user 80 performs a bending operation on the tip portion 210. In addition, at this time, the feedback control unit 130 may perform control so that an alert is presented to the user 80 via the display device 50 or the like.
[0114] Furthermore, the bending of the tip portion 210 may be restricted by controlling the operation of an attachment attached to the controller 220, for example.
[0115] The attachment may be configured to, for example, limit the rotation of a knob used to adjust the bending of the distal end portion 210 of the endoscope 20 in the controller 220 of the endoscope 20 .
[0116] Referring again to FIG. 7, the description of the example configuration of the feedback device 40 will continue.
[0117] As shown in FIG. 7, the feedback device 40 according to this embodiment includes a gate switch 430 .
[0118] FIG. 7 illustrates an example in which the feedback device 40 includes gate switches 430A and 430B.
[0119] The gate switches 430A and 430B are configured to detect the entry of the endoscope 20 into the chamber 30 and the exit of the endoscope 20 from the chamber 30.
[0120] 7 shows a state in which the distal end portion 210 of the endoscope 20 is entering the chamber 30. In this case, the gate switches 430A and 430B are activated by being pressed by the cable portion 230 of the endoscope 20.
[0121] The tracking unit 120 according to this embodiment starts tracking processing when the gate switches 430A and 430B become active.
[0122] On the other hand, when the endoscope 20 exits the chamber 30, the gate switches 430A and 430B are released from being pressed by the cable portion 230 and become inactive.
[0123] The tracking unit 120 according to this embodiment may terminate the tracking process when the gate switches 430A and 430B become inactive.
[0124] As shown in FIG. 7, the feedback device 40 according to this embodiment includes a rotation feedback structure 420 .
[0125] The rotation feedback structure 420 according to this embodiment is a configuration that corresponds to the throat, which is the passageway of the endoscope 20 during actual surgery and examination.
[0126] Portions of the throat are quite narrow, resulting in friction between the endoscope 20 and the inner walls of the throat.
[0127] The rotation feedback structure 420 according to this embodiment tactilely reproduces the friction that occurs when the endoscope 20 is rotated.
[0128] FIG. 9 is a diagram for explaining the operation of the rotation feedback structure 420 according to this embodiment.
[0129] The feedback control unit 130 according to this embodiment may change the size of the inner diameter of the rotation feedback structure 420 as shown in FIG. 9 by sending a control signal to the feedback device 40 .
[0130] According to the above control, the user 80 can get a tactile sensation when the endoscope 20 passes through the throat.
[0131] (Texture Conversion Processing) Next, the texture conversion processing according to this embodiment will be described.
[0132] As described above, the interface control unit 140 according to this embodiment controls the presentation of an image related to the three-dimensional model 60 of a biological organ according to the three-dimensional position of the endoscope 20 in the virtual space.
[0133] At this time, the interface control unit 140 may perform texture conversion processing on the texture image stored in the model database 150 in association with the three-dimensional position information, and then display it on the display device 50 .
[0134] The texture conversion process according to this embodiment may be performed using, for example, a machine learning model that has undergone codebook learning to reproduce an input image.
[0135] FIG. 10 is a diagram for explaining codebook learning according to this embodiment.
[0136] The codebook learning according to this embodiment is a type of learning that reconstructs an input image based on the encoded features of the input image.
[0137] Codebook learning proceeds by updating the network weights to minimize the reconstruction loss, an evaluation function related to the difference between the input image and the output image (reconstructed image).
[0138] The codebook training according to this embodiment uses an encoder 71, a quantizer 72, and a decoder 73.
[0139] The encoder 71 is configured to encode an input image into low-dimensional features (for example, features expressed as a vector of dimensions D, each of which is 32×32).
[0140] The input image according to this embodiment is an image FI of a living organ captured during an actual surgery or examination. The image may be, for example, a frame image extracted from a reference video.
[0141] By learning using the image FI as the input image, it becomes possible to generate an image that has been converted into a texture similar to that of an actual surgical video while maintaining the composition and other aspects when the CG is given as the input image after learning.
[0142] The quantizer 72 replaces the D-dimensional vector encoded by the encoder 71 with the most similar vector from a set of similar D-dimensional vectors registered in the codebook latent 74 .
[0143] The codebook latent 74 can be said to be a dictionary that stores a set of the D-dimensional vectors.
[0144] The decoder 73 is configured to receive the feature amount after the replacement process as an input and output a reproduced image RO that reproduces the input image.
[0145] According to the codebook learning of this embodiment, it is possible to generate a codebook latent 74 that realizes texture transformation similar to that of an actual surgical video.
[0146] FIG. 11 is a flowchart showing the flow of codebook learning according to this embodiment.
[0147] In the example shown in FIG. 11, first, a reference video is acquired (S201).
[0148] Next, frames are extracted from the reference video acquired in step S201 (S202).
[0149] Next, codebook learning is performed using the frame acquired in step S202 as an input image (S203).
[0150] Next, the codebook latent 74 generated in the codebook learning in step S203 is stored in the codebook database 160.
[0151] As described above, according to the codebook learning of this embodiment, it is possible to present to the user an image that realistically reproduces the texture and the like of an actual biological organ.
[0152] However, the texture transformation according to this embodiment does not necessarily have to be performed by codebook learning.
[0153] The texture conversion according to this embodiment may be performed using any technique capable of converting the texture into a texture similar to that of an actual surgical video. One example of such a technique is learning using a Generative Adversarial Network (GAN).
[0154] Furthermore, the interface control unit 140 according to this embodiment may also process the images to be presented to show damage (for example, bleeding), lesions (for example, cancer), and the like.
[0155] FIG. 12 is a diagram for explaining the processing according to this embodiment.
[0156] The feedback control unit 130 according to this embodiment may add a partial image P2 relating to damage, lesions, etc. to the original image P1 based on position information of an arbitrary region of the original image P1, and generate a presentation image P3.
[0157] The partial images may be provided together with the three-dimensional model 60 of the biological organ, or may be generated automatically.
[0158] (Processing Flow During Training) Next, the processing flow during training will be described.
[0159] First, the flow of loading the three-dimensional model 60 of a biological organ will be described with reference to FIG.
[0160] First, an arbitrary three-dimensional model 60 is selected by an operation by the user 80, a trainer, a scenario creator, or the like (S301).
[0161] Next, the user 80, trainer, scenario creator, or the like determines whether or not to change the texture (S302).
[0162] For example, if the user 80 wants to train by changing the texture of the three-dimensional model 60 of a biological organ, the user 80 selects a texture change. By making texture selection easy in this way, the user 80 can train with three-dimensional models 60 of various textures, which is expected to enhance the effect of improving skills.
[0163] If texture change is selected (S302: YES), the above-described texture conversion is performed (S303).
[0164] On the other hand, if a texture change is not selected (S302: NO), the default texture is loaded (S304).
[0165] Following step S303 or S304, texture styling is performed (S305).
[0166] Next, the scaling process described above is performed (S306).
[0167] Next, the three-dimensional model 60 that has been subjected to the scaling process in step S306 is loaded into memory (S307).
[0168] Next, the flow of starting and ending the tracking process according to this embodiment will be described with reference to the flowchart of FIG.
[0169] First, the user 80 inserts the endoscope 20 (S401).
[0170] Next, the tracking unit 120 determines whether the gate switch 430 is active (S402).
[0171] The tracking unit 120 repeatedly performs the determination in step S402 until it detects that the gate switch 430 is active.
[0172] When the tracking unit 120 detects that the gate switch 430 is active, it starts the tracking process (S403).
[0173] The tracking unit 120 ends the tracking process when it detects that the gate switch 430 has become inactive, or when an end operation is performed by the user 80 or the like (S404).
[0174] Next, the flow of the tracking process and feedback control according to this embodiment will be described with reference to the flowchart of FIG.
[0175] First, the tracking unit 120 acquires a frame from the endoscope 20 (S501).
[0176] The tracking unit 120 estimates the three-dimensional position of the endoscope 20 in the real space / virtual space based on the frames acquired in step S501 (S502).
[0177] Next, the tracking unit 120 performs collision detection based on the three-dimensional position of the endoscope 20 in the virtual space estimated in step S502 (S503).
[0178] If no collision is detected (S503: NO), the above-described processes related to the bio-event are carried out (S506).
[0179] On the other hand, if a collision is detected (S503: YES), the model processing unit 110 performs the above-described deformation process of the three-dimensional model 60 (S504).
[0180] Furthermore, the feedback control unit 130 performs feedback control related to collisions (S505).
[0181] Next, a description will be given of the flow of feedback by the feedback device 40 according to this embodiment. Fig. 16 is a flowchart showing the flow of feedback by the feedback device 40 according to this embodiment.
[0182] The feedback device 40 repeatedly determines whether a control signal has been received from the information processing device 10 (S601) until a control signal is received.
[0183] When the feedback device 40 receives a control signal from the information processing device 10 (S601: YES), the feedback device 40 executes feedback in accordance with the received control signal (S602).
[0184] The functions of the information processing device 10 according to this embodiment have been described in detail above.
[0185] In the above description, an endoscope has been used as an example of a medical tool, but the information processing device 10 can also track the three-dimensional position of a medical tool equipped with an imaging device other than an endoscope.
[0186] Furthermore, even if the medical tool to be tracked does not have an imaging device attached, the medical tool can be tracked if the relative position of the medical tool and a separately provided imaging device can be obtained.
[0187] Furthermore, the technical concept of this embodiment can be applied to fields other than medicine. The technical concept of this embodiment is a technology that relays the three-dimensional position of a predetermined object in real space to a three-dimensional position in virtual space, and feeds back the interaction between a virtual object in the virtual space and the predetermined object.
[0188] As an example, the technical concept of this embodiment can be applied to drone piloting and training.
[0189] In this case, a unique tag 35 is placed in a specific space in real space (which may be a closed space), and the three-dimensional position of the drone being piloted by the user within that specific space is estimated.
[0190] The user wears VR (Virtual Reality) glasses or the like, and can view the three-dimensional position of the drone in a virtual space corresponding to the specified space, as well as receive feedback regarding the drone's interaction with virtual objects.
[0191] 2. Hardware Configuration Example Next, a hardware configuration example of an information processing device 90 according to an embodiment of the present disclosure will be described. Fig. 17 is a block diagram showing a hardware configuration example of the information processing device 90 according to an embodiment of the present disclosure. The information processing device 90 may be a device having a hardware configuration equivalent to that of the information processing device 10.
[0192] 17 , the information processing device 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the information processing device 90 may include further components other than those shown here.
[0193] (Processor 871) The processor 871 functions, for example, as an arithmetic processing device or control device, and controls the overall operation of each component or part of it based on various programs recorded in the ROM 872, RAM 873, storage 880, or removable storage medium 901.
[0194] (ROM 872, RAM 873) The ROM 872 is a means for storing programs to be read into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores, for example, the programs to be read into the processor 871 and various parameters that change as appropriate when the programs are executed.
[0195] (Host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 that is capable of high-speed data transmission. On the other hand, the host bus 874 is connected to, for example, an external bus 876 that has a relatively low data transmission speed via a bridge 875. Furthermore, the external bus 876 is connected to various components via an interface 877.
[0196] (Input Device 878) For example, a mouse, keyboard, touch panel, button, switch, lever, etc. are used as the input device 878. Furthermore, a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves may also be used as the input device 878. The input device 878 also includes an audio input device such as a microphone.
[0197] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimulation.
[0198] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.
[0199] (Drive 881) The drive 881 is a device that reads information recorded on a removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901.
[0200] (Removable storage medium 901) The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable storage medium 901 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.
[0201] (Connection Port 882) The connection port 882 is a port for connecting an external device 902, such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal.
[0202] (Externally Connected Device 902) The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.
[0203] (Communication device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.
[0204] 3. Summary As described above, the information processing device 10 according to an embodiment of the present disclosure includes a model processing unit 110 that performs scaling processing to associate three-dimensional position information in the real space with three-dimensional position information of the three-dimensional model 60 of the biological organ in the virtual space, a tracking unit 120 that tracks the three-dimensional position of a medical tool operated by a user in the real space, and a feedback control unit 130 that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model 60 of the biological organ, based on the three-dimensional position of the tracked medical tool.
[0205] According to the above configuration, it is possible to eliminate restrictions on the medical tools used in training and provide a more realistic training experience.
[0206] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.
[0207] Furthermore, the steps of the processes described in this disclosure do not necessarily have to be processed in chronological order according to the order shown in the flowcharts or sequence diagrams. For example, the steps of the processes of each device may be processed in an order different from the order shown, or may be processed in parallel.
[0208] Furthermore, the series of processes performed by each device described in this disclosure may be realized by a program stored in a non-transitory computer-readable storage medium. Each program is, for example, loaded into RAM when executed by a computer and executed by a processor such as a CPU. The storage medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The program may also be distributed, for example, via a network, without using a storage medium.
[0209] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.
[0210] The following configurations also fall within the technical scope of the present disclosure. (1) An information processing device comprising: a model processing unit that performs scaling processing to associate three-dimensional position information in a real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding an interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool. (2) The information processing device described in (1), wherein the tracking unit estimates the three-dimensional position of the medical tool in the virtual space based on the three-dimensional position of the medical tool in the real space and the result of the scaling processing. (3) The information processing device described in (2), wherein the tracking unit estimates the three-dimensional position of the medical tool in the real space based on feature quantities of a unique tag placed in the real space. (4) The information processing device according to (3), wherein the tracking unit estimates a three-dimensional position of the medical tool in the real space based on a change in a feature amount of the unique tag between a plurality of images captured in time series by the medical tool. (5) The information processing device according to (2) or (3), wherein the tracking unit detects an interaction between the medical tool and a three-dimensional model of the biological organ in the virtual space. (6) The information processing device according to (5), wherein the interaction includes a collision between the medical tool and the three-dimensional model of the biological organ. (7) The information processing device according to (6), wherein the model processing unit performs a deformation process on the three-dimensional model of the biological organ when the collision is detected. (8) The information processing device according to (6) or (7), wherein the model processing unit determines whether a bio-event caused by the collision has occurred when the collision is detected. (9) The information processing device according to any one of (6) to (8), wherein the feedback control unit controls presentation of feedback corresponding to the collision when the collision is detected.(10) The information processing device according to (9), wherein the feedback corresponding to the collision includes a movement restriction of the medical tool in the real space. (11) The information processing device according to any one of (2) to (10), further comprising: an interface control unit that controls at least the presentation of an image related to a three-dimensional model of the biological organ according to a three-dimensional position of the medical tool in the virtual space. (12) The information processing device according to (11), wherein the interface control unit performs texture conversion of CG related to the three-dimensional model of the biological organ. (13) The information processing device according to (12), wherein the interface control unit performs the texture conversion using a machine learning model that reproduces an input image. (14) The information processing device according to (13), wherein the input image is an image of the biological organ captured in the real space. (15) The information processing device according to any one of (1) to (14), wherein the medical tool includes an endoscope. (16) The information processing device according to (15), wherein the endoscope is inserted into a chamber disposed in the real space. (17) An information processing method, including: a processor performing a scaling process to associate three-dimensional position information of the real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; tracking the three-dimensional position of a medical tool operated by a user in the real space; and controlling presentation of feedback regarding an interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool. (18) A program that causes a computer to function as an information processing device, comprising: a model processing unit that performs scaling processing to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
[0211] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 110 Model processing unit 120 Tracking unit 130 Feedback control unit 140 Interface control unit 20 Endoscope 30 Chamber 40 Feedback device 50 Display device 60 Three-dimensional model
Claims
1. An information processing device comprising: a model processing unit that performs a scaling process to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
2. The information processing device according to claim 1, wherein the tracking unit estimates the three-dimensional position of the medical tool in the virtual space based on the three-dimensional position of the medical tool in the real space and the result of the scaling process.
3. The information processing device according to claim 2, wherein the tracking unit estimates the three-dimensional position of the medical tool in the real space based on features of a unique tag placed in the real space.
4. The information processing device according to claim 3, wherein the tracking unit estimates the three-dimensional position of the medical tool in the real space based on changes in the features of the unique tag between multiple images taken in time series by the medical tool.
5. The information processing device according to claim 2, wherein the tracking unit detects an interaction between the medical tool and a three-dimensional model of the biological organ in the virtual space.
6. The information processing device according to claim 5, wherein the interaction includes a collision between the medical tool and a three-dimensional model of the biological organ.
7. The information processing device according to claim 6, wherein the model processing unit performs a transformation process on the three-dimensional model related to the biological organ when the collision is detected.
8. The information processing device according to claim 6, wherein the model processing unit, when the collision is detected, determines whether or not a bio-event caused by the collision has occurred.
9. The information processing device according to claim 6, wherein the feedback control unit controls the presentation of feedback corresponding to the collision when the collision is detected.
10. The information processing device according to claim 9, wherein the feedback corresponding to the collision includes a movement restriction of the medical tool in the real space.
11. The information processing device according to claim 2, further comprising: an interface control unit that at least controls the presentation of an image relating to the three-dimensional model of the biological organ according to the three-dimensional position of the medical tool in the virtual space.
12. The information processing device according to claim 11, wherein the interface control unit performs texture conversion of CG related to the three-dimensional model of the biological organ.
13. The information processing device according to claim 12, wherein the interface control unit performs the texture conversion using a machine learning model that reproduces an input image.
14. The information processing device according to claim 13, wherein the input image is an image of the biological organ captured in the real space.
15. The information processing device of claim 1, wherein the medical tool includes an endoscope.
16. The information processing device according to claim 15, wherein the endoscope is inserted into a chamber disposed in the real space.
17. An information processing method comprising: a processor performing a scaling process to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; tracking the three-dimensional position of a medical tool operated by a user in the real space; and controlling the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
18. A program that causes a computer to function as an information processing device comprising: a model processing unit that performs scaling processing to associate three-dimensional position information in real space with three-dimensional position information of a three-dimensional model of a biological organ in a virtual space; a tracking unit that tracks the three-dimensional position of a medical tool operated by a user in the real space; and a feedback control unit that controls the presentation of feedback regarding the interaction between the medical tool and the three-dimensional model of the biological organ based on the tracked three-dimensional position of the medical tool.
Citation Information
Patent Citations
Thoracoscope operation teaching system based on virtual reality
CN115662234A
Dual side cooling system for power semiconductor
KR1020240154944A
Endoscope training system
WO2019008737A1
