Information processing apparatus, display method, and program

Abstract

To provide an information processing apparatus, a display method and a program capable of checking the position of an inserted suture needle.SOLUTION: An information processing apparatus includes an image receiving unit that receives an input of an image in which a distal end portion of a suture needle is inserted into an organ, an image generating unit that generates an image in which the distal end portion is visualized by inputting the image to a machine learning model that has learned in advance, and a display control unit that causes a display apparatus to display the image in which the distal end portion is visualized.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a display method, and a program. [Background technology]

[0002] In laparoscopic surgery, doctors do not directly see the surgical site, but rather view images obtained through an endoscope, resulting in a narrow field of view. Furthermore, surgical instruments are inserted through small incisions and manipulated inside the patient's body, resulting in a small operating space. This makes laparoscopic surgery difficult for doctors, making it more prone to complications than open surgery. Some complications arise from improperly placed sutures. Once the needle is inserted, doctors rely on intuition and experience to remove the needle, making it difficult to maintain uniform suture spacing. Uneven suture spacing can make the sutures more susceptible to tearing.

[0003] For example, the following techniques have been proposed as techniques for assisting surgery. Patent Document 1 discloses acquiring information on the position and angle of a puncture instrument by attaching a position sensor and an angle sensor to the puncture instrument. Non-Patent Document 1 also discloses a technique for inputting the coordinates of the tip and rear ends of the needle at the start of insertion and the coordinates of the needle's center of gravity into a neural network model and obtaining the needle exit coordinates from the model.

[0004] Furthermore, as a technique related to the present disclosure, Non-Patent Document 2 discloses an algorithm called pix2pix, which is a type of conditional GAN ​​(Generative Adversarial Networks). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-061858 [Non-patent literature]

[0006] [Non-Patent Document 1] Masahiko Minamoto et al., “Future Needle Position Estimation of Suturing Operation using Deep Learning”, Proceedings of 2022 IEEE International Conference on Mechatronics and Automation, August 2022. [Non-patent document 2] Phillip Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks”, arXiv preprint arXiv:1611.07004, November 2016. Summary of the Invention [Problem to be solved by the invention]

[0007] The technology disclosed in Patent Document 1 is directed to a puncture instrument. A puncture instrument is not an instrument that is handled while being grasped by forceps, whereas a suture needle that is grasped and handled by forceps changes position and posture relative to the forceps. For this reason, even if a position sensor and an angle sensor were attached to the forceps instead of the puncture instrument, the position of the suture needle could not be detected. If it were possible to attach a sensor to the suture needle, it might be possible to detect the position of the suture needle, but due to the small size of the suture needle, it is difficult to attach a sensor to the suture needle.

[0008] Furthermore, the technology disclosed in Non-Patent Document 1 has a problem in that it is not possible to estimate the needle exit coordinates if the needle direction shifts after insertion because the position of the needle that has been inserted and hidden in the organ is unknown. Furthermore, although the technology in Non-Patent Document 1 estimates the needle exit coordinates with high accuracy on a fixed plane, it is unclear whether it can be estimated appropriately when an actual organ is used.

[0009] Therefore, an object of the present disclosure is to provide an information processing device, a display method, and a program that enable confirmation of the position of an inserted suture needle. [Means for solving the problem]

[0010] The information processing device according to the present disclosure includes: an image receiving unit that receives an input of an image of the tip of the suture needle being inserted into an organ; an image generation unit that generates an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control unit that displays an image in which the tip portion is visualized on a display device; It has.

[0011] In the display method according to the present disclosure, An image of the tip of the suture needle inserted into the organ is received, generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; An image in which the tip portion is visualized is displayed on a display device.

[0012] The program according to the present disclosure is an image receiving step of receiving an input of an image of the tip of the suture needle inserted into the organ; an image generation step of generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control step of displaying an image in which the tip portion is visualized on a display device; to be executed by the computer. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to provide an information processing device, a display method, and a program that enable confirmation of the position of an inserted suture needle. [Brief explanation of the drawings]

[0014] [Figure 1]1 is a block diagram showing an example of a configuration of a display system according to a first embodiment. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of a configuration of a model for obtaining a machine learning model used by an image generation unit. [Figure 3] FIG. 10 is a diagram showing an example of an image for creating a sample used in the learning phase. [Figure 4] FIG. 10 is a diagram showing an example of a sample input image used in the learning phase. [Figure 5] FIG. 10 is a diagram showing an example of a sample of a real image used in the learning phase. [Figure 6] 4 is a flowchart showing an example of an operation of the information processing device according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the configuration of a display system according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a processed image generated by an image processing unit. [Figure 9] 10 is a flowchart illustrating an example of an operation of the information processing device according to the second embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an example of a configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described with reference to the drawings. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In addition, the same elements in each drawing are designated by the same reference numerals, and duplicate explanations have been omitted as necessary.

[0016] <First Embodiment> 1 is a block diagram showing an example of the configuration of a display system 10 according to the first embodiment. The display system 10 is a system that assists a user (e.g., a doctor) in a suturing procedure, and includes an information processing device 100, a camera 200, and a display device 300, as shown in FIG.

[0017] The camera 200 is electrically connected to the information processing device 100, and images captured by the camera 200 can be input to the information processing device 100. The camera 200 may be a camera provided in an endoscope. The camera 200 includes, for example, various lenses, an imaging sensor, a signal processing circuit, and the like. As the imaging sensor, for example, a sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) is used. The images captured by the camera 200 are transmitted to the information processing device 100. The camera 200 continuously captures images at a predetermined frame rate. That is, the camera 200 captures moving images. Note that the camera 200 may also capture still images.

[0018] In this embodiment, camera 200 captures an image of suturing using a surgical suture needle. The suture needle is grasped and handled with forceps, and is repeatedly inserted into and removed from an organ to suture the target. Therefore, camera 200 can capture an image of the tip of the suture needle (i.e., a portion of the tip) inserted into the organ.

[0019] Display device 300 is electrically connected to information processing device 100 and displays an image based on the display control of information processing device 100. Display device 300 may be any known display capable of displaying an image, such as a liquid crystal display, a plasma display, or an organic EL (Electro-Luminescence) display. A user (doctor) grasps a suture needle with forceps and performs a suturing procedure while viewing, on display device 300, an image generated by information processing device 100 based on an image captured by camera 200.

[0020] Next, the information processing device 100 will be described. As shown in Fig. 1, the information processing device 100 has an image receiving unit 101, an image generating unit 102, a display control unit 103, a model learning unit 104, and a model storage unit 105. In the configuration shown in Fig. 1, the information processing device 100 includes the model learning unit 104, but the model learning unit 104 may be realized by another device. In other words, the learning process described below may be executed by a device other than the information processing device 100.

[0021] The image receiving unit 101 receives input of an image. In particular, in this embodiment, the image receiving unit 101 receives input of an image in which the tip of a suturing needle is inserted into an organ. When the information processing device 100 supports a suturing technique performed in laparoscopic surgery, the image receiving unit 101 receives an image captured by the camera 200 of the endoscope. However, the image received by the image receiving unit 101 does not necessarily have to be an image captured by the camera 200 of the endoscope. In other words, the image receiving unit 101 may receive an image captured by any camera that captures an image of a suturing target to support practice of the suturing technique, or may receive an image captured by any camera that captures an image of a suturing target to support surgery other than laparoscopic surgery, such as open surgery.

[0022] The image generation unit 102 generates an image in which the tip portion of the suture needle is visualized from an image in which the tip portion of the suture needle is inserted into an organ, that is, an image in which the tip portion of the suture needle is hidden, received by the image receiving unit 101. The image generation unit 102 generates an image in which the tip portion is visualized by inputting the image received by the image receiving unit 101 into a machine learning model that has been trained in advance and is stored in the model storage unit 105. That is, the image generation unit 102 obtains an image in which the tip portion is visualized, which is output from the machine learning model.

[0023] FIG. 2 is a schematic diagram showing an example of the configuration of a model M for obtaining a machine learning model used by the image generation unit 102. The model M is configured by a generative adversarial network (GAN). In this embodiment, more specifically, the model M is a model that implements Pix2Pix (see Non-Patent Document 2), which is a type of conditional GAN. As shown in FIG. 2, the model M is a model configured by a generator G, which is a neural network that generates an image, and a discriminator D, which is a neural network that discriminates whether an input image is a real image or a fake image generated by the generator G. In this embodiment, the generator G corresponds to a machine learning model that generates an image in which the tip portion is visualized.

[0024] An input image 51 is input to the generator G, which generates an output image 52. In the learning phase, a sample, which will be described later, is input to the generator G as the input image 51, and in the inference phase, an image captured by the camera 200 is input as the input image 51. A real image 53 as the ground truth or an output image 52, which is a fake image generated by the generator G, is input to the classifier D, which determines whether the input image is a real image or an image generated by the generator G, and outputs the determination result.

[0025] The model learning unit 104 learns the generator G using the model M, and the learned generator G is stored in the model storage unit 105. In the inference phase, only the trained generator G of the model M is used, so the model storage unit 105 only needs to store the trained generator G. However, the model storage unit 105 may also store the discriminator D used for training the generator G together with the generator G. In other words, the model storage unit 105 may store the model M.

[0026] The model training unit 104 performs processing to generate a trained machine learning model. That is, the model training unit 104 performs processing of the training phase of model M. In the training phase of model M, training of generator G and training of classifier D are repeated in order. When training generator G, the model training unit 104 calculates a loss function G Loss for generator G based on the output of classifier D, and updates the parameters of generator G based on the calculation result. Furthermore, when training classifier D, the model training unit 104 calculates a loss function D Loss for classifier D based on the output of classifier D, and updates the parameters of classifier D based on the calculation result.

[0027] In this embodiment, samples prepared as follows are used as the sample input image 51 and the sample real image 53 used in the learning phase. The sample input image 51 (see FIG. 4) and the sample real image 53 (see FIG. 5) used in the learning phase were created by the inventors by processing, using image editing software, an image 54 showing a suture needle 80 with its tip 81 not hidden against a background of an organ 90 as shown in FIG. 3. More specifically, image 54 shown in FIG. 3 shows, as an example, a suture needle 80 grasped by forceps 70.

[0028] Fig. 4 is a diagram showing an example of a sample of input image 51 used in the learning phase. As shown in Fig. 4, input image 51 used in the learning phase is obtained by processing image 54, which shows the tip and non-tip portions of a suture needle as shown in Fig. 3, by covering tip portion 81 with a drawing that resembles an organ 90.

[0029] Fig. 5 is a diagram showing an example of a sample of real image 53 used in the learning phase. As shown in Fig. 5, real image 53 used in the learning phase is obtained by processing image 54, which shows the tip and non-tip portions of a suture needle as shown in Fig. 3, by coloring tip portion 81 with a predetermined color (for example, yellow).

[0030] Thus, in this embodiment, the machine learning model used by the image generation unit 102 is a model trained by a generative adversarial network, in which an image in which the tip of the suture needle is hidden is used as an input image 51 to the generator G, and an image in which the tip of the suture needle is drawn is used as a real image 53 to the classifier D.

[0031] The model M is trained using an image in which the tip portion is rendered and visualized in a predetermined color as a real image. When an image of the tip portion of a suture needle inserted into an organ is input, the generator G outputs an image in which the tip portion is rendered and visualized in a predetermined color. Therefore, the image generation unit 102 inputs an image of the tip portion of the suture needle inserted into an organ to the generator G, thereby generating an image in which the hidden tip portion is rendered in a predetermined color. That is, in the inference phase, the image generation unit 102 inputs the image of the tip portion of the suture needle inserted into an organ, which is received by the image receiving unit 101, to the generator G, and acquires an image in which the tip portion is rendered in a predetermined color from the generator G. The image acquired in the inference phase is an image in which the position of the hidden suture needle is estimated and rendered. The images received by the image receiving unit 101 may be frame images of a video. In this case, the image generation unit 102 performs continuous processing on each frame image.

[0032] The display control unit 103 performs processing to display the image on the display device 300. In particular, the display control unit 103 causes the display device 300 to display an image in which the tip portion of the suture needle is visualized, based on the image generated by the image generation unit 102. In this embodiment, the display control unit 103 outputs the image in which the tip portion of the suture needle is visualized, generated by the image generation unit 102, to the display device 300. As described above, when the images received by the image receiving unit 101 are frame images of a moving image, the image generation unit 102 performs continuous processing on each frame image, and the display control unit 103 causes the display device 300 to display the moving image in which the tip portion is visualized. This allows the user to check the position of the suture needle in real time.

[0033] Next, the flow of operations of the information processing device 100 using the trained machine learning model will be described with reference to the flowchart shown in FIG.

[0034] In step S10, the image receiving unit 101 receives input of an image captured by the camera 200. In particular, in this embodiment, the image generating unit 102 receives input of an image in which the tip of a suture needle is inserted into an organ. Next, in step S11, the image generating unit 102 inputs the image received in step S10 into a machine learning model that has been trained in advance, thereby generating an image in which the tip portion is visualized. Next, in step S12, the display control unit 103 causes the display device 300 to display the image in which the tip portion generated in step S11 is visualized. As a result, an image in which the tip portion that should be hidden is drawn in a predetermined color, such as the output image 52 in Fig. 2, is displayed on the display device 300. That is, an image in which the hidden tip portion is filled in with a predetermined color is displayed.

[0035] The first embodiment has been described above. According to the information processing device 100, an image of the tip of a suture needle inserted into an organ is input to a machine learning model, an image in which the tip is visualized is generated, and the generated image is displayed on the display device 300. Therefore, a user who views the image displayed on the display device 300 can easily confirm the position of the inserted suture needle. In other words, the position of a hidden suture needle that would normally be invisible can be confirmed. By visualizing the needle tip hidden within an organ, even a user who is not skilled in suturing techniques can perform appropriate suturing. This is expected to reduce the occurrence of complications caused by improper suturing.

[0036] Here, we will explain the results of an experiment conducted using the machine learning model trained by the above-mentioned method. The experiment was conducted using a computer with the following specifications. Model number:HPCT W117gs GPU (Graphics Processing Unit): NVIDIA GA102 [GeForce RTX 3090] CPU (Central Processing Unit): Intel Xeon W-2245, 3.90GHz Memory:DDR4×8 (8192MB, 2933MT / s) Power supply:850 W Multi-output Power Supply Gold Certified OS(Operating System):Ubuntu 18.04.5 LTS (Bionic Beaver)

[0037] When 300 images of a suture needle tip inserted into an organ were sequentially input into a machine learning model (generator G), the inference speed, i.e., the image generation speed, of the machine learning model was 33.4 frames per second (fps). Because this is faster than a practical frame rate (e.g., 25 fps), the input video was converted in real time into a video depicting the suture needle tip. Furthermore, the average deviation between the position of the suture needle tip identified from the images generated by the machine learning model and the actual tip position was 1.03 mm. This length is less than 10% of the total length of the suture needle used in the experiment. Furthermore, the deviation was less than 3 mm in 290 of the 300 images (96.7%), and less than 1 mm in 179 of the images (59.7%). According to a report in Non-Patent Document 1, in terms of the accuracy of estimating the needle exit coordinate, deviations of less than 3 mm were achieved in 94.5% of all samples, and deviations of less than 1 mm were achieved in 49.0% of all samples. This also shows that the experimental results described above demonstrate high estimation accuracy.

[0038] As described above, in this embodiment, in the model M, the generator G is trained using an image in which the tip portion is drawn in a predetermined color as the real image 53. As a result, a generator G that generates an image in which the tip portion is drawn in the predetermined color is created. However, the image used as the real image 53 in the learning phase does not necessarily have to be an image in which the tip portion is drawn in a predetermined color; any image in which the tip portion is visualized may be used. That is, any generator G that generates an image in which the tip portion is visualized may be created. For example, an image in which the outline of a hidden tip portion is drawn using a dashed line or the like may be used as the real image 53 in the learning phase. That is, the generator G may generate an image in which the tip portion is visualized by drawing the outline of the hidden tip portion using a dashed line or the like. However, by using an image in which the hidden tip portion is clearly drawn as shown in FIG. 5, high estimation accuracy can be achieved, as shown in the experimental results described above. That is, it is preferable that the generator G (image generating unit 102) generates an image in which the tip portion is drawn in a predetermined color as the image in which the tip portion is visualized. Furthermore, by drawing the tip portion in a predetermined color, the user can easily identify the tip portion in the image.

[0039] <Embodiment 2> Next, a description will be given of embodiment 2. In embodiment 1, the display control unit 103 causes the display device 300 to display the image generated by the image generation unit 102, but in this embodiment, the display control unit 103 displays an image generated by image processing using the image accepted by the image acceptance unit 101 and the image generated by the image generation unit 102.

[0040] 7 is a block diagram showing an example of the configuration of a display system 10a according to the second embodiment. The display system 10a according to the second embodiment differs from the display system 10 according to the first embodiment in that the information processing device 100 is replaced with an information processing device 100a. The information processing device 100a also differs from the information processing device 100 in that it further includes an image processing unit 106. Below, configurations and processes that differ from those of the first embodiment will be described, and overlapping descriptions will be omitted as appropriate.

[0041] The image processing unit 106 performs image processing using the input image 51, which is an image received by the image receiving unit 101, and the image in which the tip of the suture needle is visualized, which is the output image 52 from the machine learning model (generator G), to generate an image in which the tip is visualized and an organ in the area where the tip is located is depicted. Hereinafter, the image generated by the processing of the image processing unit 106 will be referred to as a processed image. Specifically, the image processing unit 106 generates a processed image 55 as shown in FIG. 8 using, for example, the input image 51 and output image 52 shown in FIG. 2. In the output image 52 shown in FIG. 2, the tip of the suture needle is filled in with a predetermined color, but in the processed image 55 shown in FIG. 8, only an outline 81a of the tip of the suture needle is drawn, and the organ is depicted inside the outline 81a. The image processing unit 106 generates the processed image 55 as follows. For example, the image processing unit 106 extracts the outline of a region filled with a predetermined color in the output image 52 (i.e., the outline of the hidden tip portion) and draws the extracted outline on the input image 51, thereby generating the processed image 55. That is, the image processing unit 106 generates the processed image 55 by overlaying the extracted outline on the input image 51. Note that in the processed image 55, the outline may be drawn using a solid line or a dashed line. Furthermore, for example, the image processing unit 106 may extract the region filled with a predetermined color in the output image 52 (i.e., the region corresponding to the hidden tip portion) and draw the extracted region in a semi-transparent color on the input image 51, thereby generating the processed image.

[0042] In this embodiment, the display control unit 103 displays the processed image generated by the image processing unit 106 on the display device 300. With this configuration, as shown in Fig. 8, it is possible to visualize the tip of the suture needle and prevent the drawing for visualizing the tip from obscuring the visibility of an organ that is actually visible.

[0043] Next, the flow of operations of the information processing device 100a according to this embodiment will be described with reference to the flowchart shown in Fig. 9. The flowchart shown in Fig. 9 differs from the flowchart shown in Fig. 6 in that step S20 is added after step S11 and step S12 is replaced with step S21. The differences from the flowchart shown in Fig. 6 will be described below.

[0044] In this embodiment, after step S11, the process proceeds to step S20. In step S20, the image processing unit 106 performs image processing using the image received in step S10 and the image generated in step S11, thereby visualizing the tip of the suture needle and generating an image showing the organs in the area where the tip is located. Next, in step S21, the display control unit 103 causes the display device 300 to display the image generated in step S20, in which the tip portion is visualized and the organ in the area where the tip portion is located is shown.

[0045] The above describes embodiment 2. According to information processing device 100a, the tip of the suture needle can be visualized, and an image of an organ in the area where the tip is located can be displayed on display device 300. This makes it possible to visualize the tip of the suture needle, and also prevents the drawing for visualizing the tip from obscuring the organ that is actually visible.

[0046] Although the first and second embodiments have been described, the following modifications may also be adopted to these embodiments. The display system 10 or the display system 10a may provide a user with a 3D (three-dimensional) image. In this modification, for example, a camera such as a stereo camera that captures an object from two different viewpoints and outputs two images is used as the camera 200. In this modification, the image receiving unit 101 receives input of two images captured from different viewpoints. That is, the image receiving unit 101 receives input of two images that are simultaneously captured from different viewpoints and show the tip of a suture needle inserted into an organ. Then, the image generating unit 102 generates an image in which the tip of the suture needle is visualized for each of the two images received by the image receiving unit 101. That is, the image generating unit 102 generates two such images by inputting each image to a machine learning model. In this modification, the display control unit 103 uses two images in which the tip of the suture needle is visualized to cause the display device 300 to display an image for stereoscopic viewing. For example, the display control unit 103 synthesizes the two images generated by the image generation unit 102 to generate a three-dimensional image. The user views the three-dimensional image displayed on the display device 300 through, for example, 3D glasses, thereby stereoscopically viewing the three-dimensional image. Note that the processing of the image processing unit 106 may also be performed in this modification. In that case, the display control unit 103 generates a three-dimensional image using the two images generated by the image processing unit 106 and displays the three-dimensional image on the display device 300. According to this modification, the user can perform the suturing procedure while stereoscopically viewing the suture needle and its surroundings, and therefore can perform the suturing procedure while grasping the sense of perspective.

[0047] The above-described functions (processing) of the information processing device 100 and the information processing device 100a may be realized by a computer 500 having the following configuration, for example.

[0048] 10 is a block diagram showing an example of the configuration of a computer 500 that realizes the processing of the information processing device 100 and the information processing device 100a. As shown in FIG. 10, the computer 500 includes an input / output interface 501, a memory 502, and a processor 503.

[0049] The input / output interface 501 is an interface for connecting to other devices (for example, the camera 200 and the display device 300).

[0050] The memory 502 is configured, for example, by a combination of volatile memory and non-volatile memory, and is used to store software (computer programs) including one or more instructions executed by the processor 503, data used for various processes, and the like.

[0051] The processor 503 performs the above-described processing of the information processing device 100 or the information processing device 100a by reading and executing software (computer programs) from the memory 502. The processor 503 may be, for example, a microprocessor, an MPU (Micro Processor Unit), a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 503 may include multiple processors.

[0052] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0053] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0054] Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. (Appendix 1) an image receiving unit that receives an input of an image of the tip of the suture needle being inserted into an organ; an image generation unit that generates an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control unit that displays an image in which the tip portion is visualized on a display device; An information processing device having the above. (Appendix 2) The machine learning model is a model trained by a generative adversarial network using an image in which the tip of the suture needle is hidden as an input image to the generator and an image in which the tip of the suture needle is drawn as a real image to the classifier. 10. The information processing device according to claim 1. (Appendix 3) The image generating unit generates an image in which the tip portion is drawn in a predetermined color. 3. The information processing device according to claim 1 or 2. (Appendix 4) an image processing unit that performs image processing using the image received by the image receiving unit and the image in which the tip portion is visualized output from the machine learning model to generate a processed image in which the tip portion is visualized and the organ in the area in which the tip portion is located is depicted; The display control unit causes the display device to display the processed image as an image in which the tip portion is visualized. 4. The information processing device according to claim 1. (Appendix 5) the image receiving unit receives input of two images captured from different photographing viewpoints; the image generation unit generates an image in which the tip portion is visualized for each of the two images, The display control unit causes the display device to display an image for stereoscopic viewing using the two images in which the tip portion is visualized. 5. The information processing device according to any one of claims 1 to 4. (Appendix 6) the image received by the image receiving unit is a frame image of a moving image, The display control unit causes the display device to display a moving image in which the tip portion is visualized. 6. An information processing device according to any one of appendices 1 to 5. (Appendix 7) An image of the tip of the suture needle inserted into the organ is received, generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; An image in which the tip portion is visualized is displayed on a display device. Display method. (Appendix 8) an image receiving step of receiving an input of an image of the tip of the suture needle inserted into the organ; an image generation step of generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control step of displaying an image in which the tip portion is visualized on a display device; A program that causes a computer to execute the following. [Explanation of symbols]

[0055] 10 Display System 10a Display System 51 input images 52 Output Images 53 Real Images 54 images 55 processed images 70 forceps 80 suture needle 81 Tip part 81a Contour 90 Organs 100 Information processing device 100a Information processing device 101 Image Reception Department 102 Image generation unit 103 Display control unit 104 Model Learning Department 105 Model Memory Unit 106 Image processing section 200 cameras 300 display device 500 computers 501 Input / Output Interface 502 memory 503 processor D Discriminator G generator M Model

Claims

1. an image receiving unit that receives an input of an image of the tip of the suture needle being inserted into an organ; an image generation unit that generates an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control unit that displays an image in which the tip portion is visualized on a display device; An information processing device having the above.

2. The machine learning model is a model trained by a generative adversarial network using an image in which the tip of the suture needle is hidden as an input image to the generator and an image in which the tip of the suture needle is drawn as a real image to the classifier. The information processing device according to claim 1 .

3. The image generating unit generates an image in which the tip portion is drawn in a predetermined color.

3. The information processing device according to claim 1 or 2.

4. an image processing unit that performs image processing using the image received by the image receiving unit and the image in which the tip portion is visualized output from the machine learning model to generate a processed image in which the tip portion is visualized and the organ in the area in which the tip portion is located is depicted; The display control unit causes the display device to display the processed image as an image in which the tip portion is visualized. The information processing device according to claim 1 .

5. the image receiving unit receives input of two images captured from different imaging viewpoints; the image generation unit generates an image in which the tip portion is visualized for each of the two images, The display control unit causes the display device to display an image for stereoscopic viewing using two images in which the tip portion is visualized. The information processing device according to claim 1 .

6. the image received by the image receiving unit is a frame image of a moving image, The display control unit causes the display device to display a moving image in which the tip portion is visualized. The information processing device according to claim 1 .

7. An image of the tip of the suture needle inserted into the organ is received, generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; An image in which the tip portion is visualized is displayed on a display device. Display method.

8. an image receiving step of receiving an input of an image of the tip of the suture needle inserted into the organ; an image generation step of generating an image in which the tip portion is visualized by inputting the image into a pre-trained machine learning model; a display control step of displaying an image in which the tip portion is visualized on a display device; A program that causes a computer to execute the following.

Images