Image processing device, image processing method, surgical system and surgical thread

Abstract

Image processing device (53) comprising: an image obtaining unit (52) which obtains a first image generated under an illumination condition in which a surgical thread (32) fluoresces and a second image generated under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread (32), a synthesis unit (77) that generates a synthetic image obtained by synthesizing a portion of the surgical thread (32) of the first image and the second image; and a color conversion processing unit (78) which converts a color of a needle (33) of the synthetic image according to an antero-posterior relationship between the surgical thread (32) and the needle (33).

Description

TECHNICAL FIELD

[0001] The present technology relates to an image processing apparatus, an image processing method, an operation system, and a surgical thread, and more particularly to the image processing apparatus, the image processing method, the operation system, and the surgical thread, whereby the visibility of the thread can be further improved. TECHNICAL BACKGROUND

[0002] For example, the visibility of a suture thread in a surgical operation and the like is improved by making a color of the suture thread different from that in a living body (for example, black or purple). However, there is a limit to improving visibility by specifying only the color due to the small diameter of the suture thread to be used and the like.

[0003] On the other hand, a technology for improving visibility by allowing the thread itself to glow is also being developed (see, for example, Patent Document 1).

[0004] Patent Document 2 relates to a suture tool kit and a method for visualizing a surgical suture. It describes a suture thread with segmented regions, each exhibiting different spectral signatures in the non-visible wavelength range. A camera captures a spectrally resolved image of the thread, while an image processor assigns high-contrast colors to make the suture structure more visible. This improves the identification of the thread arrangement, especially in the case of overlaps or when the thread is covered by body fluids. Furthermore, a depth sensing method is described to determine the spatial arrangement of the suture.

[0005] Patent Document 3 relates to a medical device with near-infrared fluorescence and a system for checking its usage status. It describes a light-emitting medical device containing a luminescent substance that emits near-infrared fluorescence when irradiated with excitation light. This allows for easier detection of damaged or lost parts, as well as small objects such as suture needles or threads. The system includes a light source, an optical filter, a camera, and a display for displaying fluorescent images. This enables precise monitoring of the device, even when it is buried under tissue or in contact with body fluids.

[0006] Patent Document 4 describes an image pickup device and method for generating a synthesized image by combining a standard image captured with visible light and a special image generated by fluorescence excitation. This technology is particularly used in medical and industrial applications. The device includes multiple image pickup systems and image processing sections that extract relevant image information and generate optimized synthesized images. Various contour and gradation correction methods improve the visibility of specific structures, such as blood vessels or diseased tissue, to enable more precise image representation. CITATION LISTPATENT DOCUMENT Patent Document 1: Japanese Patent Application Laid-Open No. JP 2010-095833 A Patent document 2: EP 2 700 365 A1 Patent Document 3: WO 2013 180 127 Patent document 4: DE 10 2014 110 131 A1 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, there is also a limit to improving visibility simply by allowing the thread itself to fluoresce.

[0008] The present technology was developed in view of such a situation, and one of its tasks is to further improve the visibility of both the thread and a needle in a synthetically combined image. SOLUTIONS TO PROBLEMS

[0009] This is achieved by devices and methods according to the independent patent claims. Advantageous further developments can be found in the dependent patent claims.

[0010] An image processing apparatus according to a first aspect of the present technology includes: an image obtaining unit that obtains a first image obtained under an illumination condition in which a surgical thread fluoresces and a second image obtained under an illumination condition including at least visible light as images of a surgical site using the fluorescent surgical thread; and a synthesis unit that generates a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image.

[0011] An image processing method according to the first aspect of the present technology comprises the steps of: obtaining a first image formed under an illumination condition in which a surgical thread fluoresces and a second image formed under an illumination condition including at least visible light as images of a surgical site using the fluorescent surgical thread; and generating a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image.

[0012] In the first aspect of the present technology, a first image formed under an illumination condition in which a surgical thread fluoresces and a second image formed under an illumination condition including at least visible light are obtained as images of a surgical site using the fluorescent surgical thread, and a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image is generated.

[0013] A surgical system according to a second aspect of the present technology includes: an imaging unit that generates a first image generated under an illumination condition in which a surgical thread fluoresces and a second image generated under an illumination condition including at least visible light as images of a surgical site using the fluorescent surgical thread; and a synthesis unit that generates a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image.

[0014] According to the second aspect of the present technology, a first image formed under an illumination condition in which a surgical thread fluoresces and a second image formed under an illumination condition including at least visible light are formed as images of a surgical site using the fluorescent surgical thread, and a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image is formed.

[0015] A surgical thread according to a third aspect of the present technology fluoresces in a pattern not present in a living body.

[0016] According to the third aspect of the present technology, it fluoresces in a pattern that is not present in a living body.

[0017] The image processing device may be an independent device or an internal block constituting a device. EFFECTS OF THE INVENTION

[0018] According to the first to third aspects of the present technology, the visibility of the thread can be further improved.

[0019] The effects are not necessarily limited to those described here and may include any of the effects described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWING

[0020] They show: Fig. 1 is a view showing a configuration example of an endoscopic surgical system to which the present technology is applied, Fig. 2 a block diagram relating to the image processing of the endoscopic surgical system, Fig. 3 a view explaining image synthesis processing, Fig. 4 is a view for explaining imaging timings for a normal light image and a special light image, Fig. 5 is a block diagram of a detailed configuration example of an image processing unit, Fig. 6 is a block diagram of a detailed configuration example of a motion vector detection unit, Fig. 7 is a flowchart explaining the image synthesis processing by the image processing unit, Fig. 8 is a view for explaining a processing flow for estimating a movement correction amount, Fig. 9 a view of an image for correcting the variation of motion vectors, Fig. 10 is a view for explaining another example of the imaging timings for the normal light image and the special light image, Fig. 11 is a view for explaining an image sensor of an imaging unit according to a second embodiment, Fig. 12 is a view for explaining image synthesis processing according to the second embodiment, Fig. 13 is a view of another configuration example of a fluorescent thread, Fig. 14 is a view of another configuration example of the fluorescent thread, Fig. 15 is a block diagram of a variation of a synthesis unit of an image processing unit, Fig. 16 is a view showing an example of an image after color conversion processing, Fig. 17 is a view for explaining another application example of the color conversion process, Fig. 18 a flowchart explaining the color conversion processing, Fig. 19 is a flowchart for explaining pixel addition processing in a region of the fluorescent filament, Fig. 20 is a view showing an example of pixel addition processing in a region of the fluorescent filament, and Fig. 21 is a block diagram of a configuration example of an embodiment of a computer to which the present technology is applied. MODE FOR CARRYING OUT THE INVENTION

[0021] Modes for implementing the present technology (hereinafter referred to as embodiments) will be described below. The description will be given in the following order. 1. First Embodiment (Example of Separate Imaging of a Special Light Image and a Normal Light Image) 2. Second Embodiment (Example of generating a special light image and a normal light image from one image) 3. Another configuration example of the fluorescent thread 4. Variation of the synthesis unit 5. Other application example of color conversion processing 6. Pixel addition processing in the fluorescent thread area <1. First Embodiment><Konfigurationsbeispiel eines endoskopischen Operationssystems>

[0022] Fig. 1 shows a configuration example of an endoscopic surgical system to which the present technology is applied.

[0023] An endoscopic surgical system 10 is provided with a carriage 18 on which a display device 11, a camera control unit (CCU) 12, a light source device 13, a treatment tool device 14, a pneumoperitoneum device 15, a recording device 16, and a printer 17 are mounted. The endoscopic surgical system 10 also includes an endoscope (laparoscope) 19, an energy treatment tool 20, a forceps 21, trocars 22 to 25, a foot switch 26, and a patient bed 27. The endoscopic surgical system 10, arranged in an operating room, for example, assists a surgeon performing laparoscopic surgery on an affected area in the abdomen 31 of a patient lying on the patient bed 27.

[0024] Specifically, the display device 11 of the endoscopic surgical system 10 includes a stationary 2D display, a head-mounted display, and the like. The display device 11 displays an image of a surgical site supplied from the CCU 12, and the like.

[0025] The CCU 12 (image processing device) is connected to the endoscope 19 via a camera cable. However, the CCU 12 may also be wirelessly connected to the endoscope 19. The CCU 12 receives an image of a site to be operated on (hereinafter also referred to as an operation site image), which is generated by the endoscope 19 and transmitted via the camera cable, and supplies it to the display device 11. The CCU 12 supplies the received operation site image to the recording device 16 and the printer 17 as needed.

[0026] The light source device 13 is connected to the endoscope 19 via a fiber optic cable. The light source device 13 switches light of different wavelengths and transmits it to the endoscope 19.

[0027] The treatment tool device 14, which is a high-frequency output device, is connected by cables to the energy treatment tool 20 and the foot switch 26. The treatment tool device 14 outputs a high-frequency current to the energy treatment tool 20 in response to an operating signal supplied from the foot switch 26.

[0028] The pneumoperitoneum device 15, which is provided with an air supply device and an air suction device, supplies air to the abdomen 31 through a hole of the trocar 24, which is a hole-forming tool attached to the abdominal wall of the abdomen 31.

[0029] The recording device 16 records the surgical site image supplied from the CCU 12. The printer 17 prints the surgical site image supplied from the CCU.

[0030] The endoscope 19 is inserted into the abdomen 31 to be operated on through a hole of the trocar 22 attached to the abdominal wall of the abdomen 31. The endoscope 19 irradiates the interior of the abdomen 31 with the light emitted by the light source device 13 and images the interior of the abdomen 31 as an operation site image. The endoscope 19 transmits the operation site image obtained by an imaging unit 41 to the CCU 12 via the camera cable.

[0031] The energy treatment tool 20 (treatment device) includes an electric scalpel and the like. The energy treatment tool 20 is inserted into the abdomen 31 through a hole of the trocar 23 attached to the abdominal wall of the abdomen 31. The energy treatment tool 20 denatures or cuts the interior of the abdomen 31 using electric heat.

[0032] The forceps 21 are inserted into the abdomen 31 through a hole of the trocar 25 attached to the abdominal wall of the abdomen 31. The forceps 21 penetrates the interior of the abdomen 31. The endoscope 19, the energy treatment tool 20, and the forceps 21 are grasped by the surgeon, an assistant, a scopist, a robot, or the like.

[0033] The foot switch 26 accepts an operation by a foot of the surgeon, assistant, etc. The foot switch 26 supplies the operating signal indicating the accepted operation to the CCU 12 and the treatment tool device 14.

[0034] By using the endoscopic surgical system 10, the surgeon can perform an excision at the affected area in the abdomen 31 without performing an abdominal operation in which the abdominal wall is cut open and the abdomen is opened.

[0035] While observing a processed image displayed on the display device 11 based on the image generated by the imaging unit 41 of the endoscope 19, the surgeon treats the affected area (operation site) to be operated in the abdomen 31. At this time, a fluorescent thread 32 is used to suture and knot the affected area. The fluorescent thread 32 is a thread that fluoresces when the illumination irradiating the affected area assumes a predetermined condition. For example, a silk thread luminescent with a green fluorescent protein, as disclosed in Patent Document 1 described above, can be used.

[0036] The fluorescent thread 32 fluoresces so that it can be easily identified visually, but the surgeon may have difficulty seeing the affected area under the lighting in which the fluorescent thread 32 fluoresces, for example, if it is dark when no visible light is used.

[0037] Therefore, the CCU 12 of the endoscopic surgical system 10 controls the light source device 13 and the endoscope 19 to perform imaging under an illumination condition in which the surgeon can easily visually recognize the fluorescent thread 32, and to perform imaging under an illumination condition in which the surgeon can easily visually recognize the affected site. The CCU 12 also performs image synthesis processing on a plurality of surgical site images obtained under a plurality of illumination conditions, thereby generating a synthetic image in which the visibility of the fluorescent thread 32 is enhanced, and allows the display device 11 to display it. The surgeon can perform surgery while observing the synthetic image displayed on the display device 11, and can smoothly suture and tie knots. <Blockdiagramm, das sich auf die Bildverarbeitung bezieht>

[0038] Fig. 2 is a block diagram focusing on a block related to imaging under the illumination condition in which the fluorescent thread 32 is easily visually recognized, imaging under the illumination condition in which the affected site is easily visually recognized, and image processing of multiple surgical site images obtained under multiple illumination conditions.

[0039] The CCU 12 is provided with at least a control unit 51, an image obtaining unit 52, an image processing unit 53 and an image output unit 54.

[0040] The control unit 51 performs illumination control for switching between the illumination under the illumination condition in which the fluorescent thread 32 can be easily visually recognized and the illumination under the illumination condition in which the affected site can be easily visually recognized, according to the timing of imaging by the imaging unit 41 of the endoscope 19 on the light source device 13.

[0041] The control unit 51 also controls the imaging timing by the imaging unit 41 of the endoscope 19. The endoscope 19 is provided with an optical system such as an illumination lens and the imaging unit 41. The imaging unit 41 includes a CMOS sensor of a Bayer array, for example, in which four pixels, namely R, G, R, and B pixels, are repeatedly arranged. The imaging unit 41 performs imaging based on the imaging timing controlled by the control unit 51 and outputs the resulting image of the affected area to the image obtaining unit 52.

[0042] Hereinafter, the illumination under the illumination condition in which the fluorescent thread 32 can be easily visually identified is referred to as special light, and the illumination under the illumination condition in which the affected area can be easily visually identified is referred to as ordinary light. The special light includes, for example, ultraviolet light called black light, infrared light (IR light), and the like. The ordinary light includes, for example, white light (visible light).

[0043] Also, an image formed by irradiation with special light is called a special light image, and an image formed by irradiation with normal light is called a normal light image.

[0044] The control unit 51 controls the switching between the special light and the normal light emitted from the light source device 13. Furthermore, the control unit 51 outputs an irradiation identification signal indicating which of the images obtained by irradiation with the normal light and the special light is supplied from the image obtaining unit 52 to the image processing unit 53.

[0045] In addition to the control described above, the control unit 51 controls the operation of the entire endoscopic operation system 10, and thereby controls, for example, the treatment tool device 14 and the pneumoperitoneum device 15 based on the operation signal supplied from the foot switch 26, as well as the recording device 16 and the printer 17.

[0046] The image obtaining unit 52 receives the normal light image and the special light image supplied from the imaging unit 41 and supplies them to the image processing unit 53.

[0047] The image processing unit 53 performs predetermined image processing on the normal light image and the special light image supplied from the image obtaining unit 52 and supplies the processed image to the image output unit 54.

[0048] The image processing unit 53 detects, for example, a motion vector using two normal light images at different imaging times. The image processing unit 53 also performs predetermined filtering processing on the special light image, thereby generating an image obtained by extracting a portion of the fluorescent filament 32 in the special light image (hereinafter referred to as an extracted fluorescent filament image). Further, based on the motion vector detected from the normal light image, the image processing unit 53 performs motion correction on the extracted fluorescent filament image, synthesizes the motion-corrected extracted fluorescent filament image and the normal light image, and outputs the resulting synthesized image to the image output unit 54.

[0049] The image output unit 54 converts the synthetic image supplied from the image processing unit 53 into a signal having a predetermined format that the display device 11 can accept and outputs it to the display device 11. Based on the image signal supplied from the image output unit 54, the display device 11 displays the generated synthetic image so that the operator can easily visually recognize the affected area and the fluorescent thread 32.

[0050] Fig. 3 conceptually shows the image synthesis processing performed by the image processing unit 53.

[0051] The normal light image is, for example, the image produced by irradiation with white light, namely the image in which the affected area can be easily identified visually, but the fluorescent thread 32 is not easily visible.

[0052] On the other hand, the special light image is, for example, the image produced by irradiation with the IR light, namely the image in which the fluorescent thread 32 can be easily recognized visually, but the affected area is not easily visible.

[0053] The synthetic image obtained by synthesizing the normal light image and the special light image is the image using the normal light image for the affected area and the special light image for the fluorescent thread 32. Therefore, both the affected area and the fluorescent thread 32 can be easily visually recognized in the images. <Bildgebungszeitablauf beim Normallichtbild und beim Spezielles-Licht-Bild>

[0054] Next, Fig. 4 an example of the imaging time sequences for the normal light image and the special light image.

[0055] The control unit 51 controls the continuous generation of multiple frames of the normal light images and then the generation of the special light image. The image generation ratio for the normal light images and the special light images is, for example, 4:1, as shown in Fig. 4. However, this ratio is not limited to 4:1 and can be changed.

[0056] The time Ta in Fig. Time 4 indicates the time at which the normal light image is generated one frame before the special light image, and time Tb indicates the time at which the special light image is generated. Times Tc, Td, Te, and Tf indicate the times at which the normal light image is generated one, two, three, and four frames after the special light image, respectively. <Detailliertes Konfigurationsbeispiel der Bildverarbeitungseinheit 53>

[0057] Fig. 5 is a block diagram showing a detailed configuration example of the image processing unit 53.

[0058] The image processing unit 53 includes a switching unit 71, a motion vector detection unit 72, a correction amount estimation unit 73, a frame memory 74, a feature extraction filter processing unit 75, a motion correction unit 76, and a synthesis unit 77.

[0059] In the image processing unit 53, the normal light image and the special light image received from the image obtaining unit 52 are input to the switching unit 71, and the irradiation identification signal from the control unit 51 is input to the switching unit 71, the motion vector detection unit 72, and the correction amount estimation unit 73.

[0060] Based on the irradiation identification signal, the switching unit 71 determines whether the input of the image obtaining unit 52 is the special light image, and if not (which is the normal light image), outputs it to the motion vector detection unit 72 and the synthesis unit 77, and if it is the special light image, outputs it to the frame memory 74.

[0061] The motion vector detection unit 72 detects the motion vector using two normal light images with different imaging times for each frame period and outputs the detected motion vector to the correction amount estimation unit 73.

[0062] Based on the motion vector detected by the motion vector detection unit 72, the correction amount estimation unit 73 estimates the motion correction amount of the special light image and the extracted fluorescent thread image, and outputs the estimated motion correction amount to the motion correction unit 76. Here, the correction amount estimation unit 73 can correct the motion vector that might be misdetected based on the continuously detected motion vectors and estimate the motion correction amount based on the corrected motion vector.

[0063] The frame memory 74 holds the special light image input from the switching unit 71 and supplies the held special light image to the feature extraction filter processing unit 75 and the motion correction unit 76 in each frame period. Even in a case where the next special light image is input from the switching unit 71, the frame memory 74 updates the held special light image.

[0064] The feature extraction filter processing unit 75 extracts a region of the fluorescent filament 32 based on the feature of the fluorescent filament 32 in the special light image supplied from the frame memory 74, and generates the extracted fluorescent filament image showing an extraction result. Specifically, the feature extraction filter processing unit 75 extracts, for example, a region within a specific area where the signal level of a pixel corresponds to a fluorescent color, that is, a region with a specific RGB value, and generates the extracted fluorescent filament image showing the extraction result.

[0065] At this time, the frame memory 74 supplies the retained special light image every frame period as described above, so that this special light image is continuously supplied. In this case, the feature extraction filter processing unit 75 can output a result of the previous feature extraction filter processing unchanged to the motion correction unit 76 while skipping the feature extraction filter processing.

[0066] The feature extraction filter processing unit 75 can perform feature extraction filter processing other than the extraction of the region with the specific RGB value described above, such as differential filter processing (e.g., Sobel filter processing), contour detection processing, variance-based processing, dynamic range processing, and the like in each small (3 x 3) block defined in the image. That is, the feature extraction filter processing executed by the feature extraction filter processing unit 75 is not limited as long as it can extract the region of the fluorescent thread 32.

[0067] Based on the motion correction amount input from the motion correction amount estimation unit 73, the motion correction unit 76 performs the motion correction of the special light image from the frame memory 74, performs the motion correction of the extracted fluorescent filament image from the feature extraction filter processing unit 75, and outputs the motion-corrected special light image and the extracted fluorescent filament image to the synthesis unit 77.

[0068] The synthesis unit 77 performs synthesis processing to synthesize the normal light image and the motion-corrected extracted image of the fluorescent thread, thereby generating the synthetic image, and outputs it to the image output unit 54. For example, the synthesis unit 77 generates the synthetic image using the normal light image for the affected area portion and the special light image for the fluorescent thread 32. Also, for example, the synthesis unit 77 generates the synthetic image by superimposing a portion of the fluorescent thread 32 in the special light image on the normal light image. In this way, the synthetic image in which both the affected area and the fluorescent thread 32 can be easily visually recognized can be generated, as shown in FIG. Fig. 3 is shown.

[0069] In addition to the normal light image and the motion-corrected extracted image of the fluorescent thread, the motion-corrected special light image is also input into the synthesis unit 77, so that the motion-corrected special light image can also be used in the synthesis processing if necessary. <Detailliertes Konfigurationsbeispiel der Bewegungsvektor-Erkennungseinheit 72>

[0070] Fig. 6 is a block diagram showing a detailed configuration example of the motion vector detection unit 72.

[0071] The motion vector detection unit 72 includes frame memories 91 and 92, a frame selection unit 93, a block comparison unit 94, and a vector correction unit 95.

[0072] In the motion vector detection unit 72, the normal light image received from the switching unit 71 in a previous stage is input into the frame memory 91 and the frame selection unit 93.

[0073] For each frame period, the frame memory 91 outputs the normal light image held up to that point to the frame memory 92 and the frame selection unit 93, and updates the held data with the normal light image received from the switching unit 71 in the previous stage. Similarly, for each frame period, the frame memory 92 outputs the normal light image held up to that point to the frame selection unit 93, and updates the held data with the normal light image received from the frame memory 91 in the previous stage.

[0074] However, in the frame period, at the time when the normal light image is not input to the motion vector detection unit 72, the frame memory 91 outputs the normal light image held until then to a subsequent stage and deletes the data held until then.

[0075] At the next time, frame memory 91 does not output any data to the next stage because there is no held data. Frame memory 92 outputs the normal light image held until then to the next stage and deletes the data held until then.

[0076] Therefore, two or three normal light images of different imaging times are simultaneously input into the frame selection unit 93.

[0077] In a case where the two normal light images are input simultaneously, the frame selection unit 93 outputs the two normal light images to the block comparison unit 94. Also, in a case where three normal light images are input simultaneously, the frame selection unit 93 outputs the two normal light images input from the frame memories 91 and 92 to the block comparison unit 94. The block comparison unit 94 detects the motion vector between the two normal light images through block comparison processing.

[0078] Based on the irradiation identification signal, the vector correction unit 95 determines a relationship between the two normal light images used for the motion vector, corrects the detected motion vector based on the relationship, and outputs the motion vector to the correction amount estimation unit 73.

[0079] The correction of the motion vector by the vector correction unit 95 will be specifically described.

[0080] With respect to the output from the frame memory 91, it should be noted that in a case where the reference imaging time is the one specified in Fig. 4, the normal light image one frame before the reference and the reference normal light image are input from the frame memory 92 and the frame memory 91, respectively, into the frame selection unit 93, and the motion vector is detected from the two normal light images. In this case, the vector correction unit 95 does not correct the motion vector.

[0081] In a case where the reference imaging time exceeds the Fig. 4 is Tb, the frame memory 91 does not output anything because the time Tb is the imaging time of the special light image. Then, the normal light image at time Ta one frame before the reference and the normal light image at time Tc one frame after the reference are input to the frame selection unit 93 from the frame memory 92 and the switching unit 71, respectively, and the motion vector is detected from the two normal light images. In this case, the vector correction unit 95 multiplies both the vertical and horizontal components of the detected motion vector by 1 / 2 because the detected motion vector is that between the normal light images separated by two frames.

[0082] In a case where the reference imaging time exceeds the Fig. 4, the reference standard light image and the standard light image at time Td are input to the frame selection unit 93 from the frame memory 91 and the switching unit 71, respectively, one frame after the reference, and the motion vector is detected from the two standard light images. Because the direction of the detected motion vector in this case is opposite to that in other cases, the vector correction unit 95 multiplies both the vertical and horizontal components of the detected motion vector by -1.

[0083] In a case where the reference imaging time exceeds the Fig. 4, the normal light image at time Tc one frame before the reference, the reference normal light image, and the normal light image at time Te one frame after the reference are input to the frame selection unit 93 from the frame memory 92, the frame memory 91, and the switching unit 71, respectively, and the motion vector is detected from the two normal light images from the frame memories 91 and 92. In this case, the vector correction unit 95 does not correct the motion vector.

[0084] In a case where the reference imaging time exceeds the Fig. 4, the normal light image at time Td one frame before the reference, the reference normal light image, and the normal light image at time Tf one frame after the reference are input to the frame selection unit 93 from the frame memory 92, the frame memory 91, and the switching unit 71, respectively, and the motion vector is detected from the two normal light images from the frame memories 91 and 92. In this case, the vector correction unit 95 does not correct the motion vector.

[0085] The motion vector corrected in the manner described above is output from the vector correction unit 95 to the correction amount estimation unit 73 in the following stage. <Flussdiagramm der Bildsyntheseverarbeitung>

[0086] The image synthesis processing by the image processing unit 53 will next be described with reference to a flowchart in Fig. 7. This image synthesis processing is performed in each frame period.

[0087] In step S1, the switching unit 71 determines whether the input from the image obtaining unit 52 is the special light image or the normal light image based on the irradiation identification signal, and switches an output destination of the input image. That is, if the switching unit 71 determines that the input from the image obtaining unit 52 is the special light image, it outputs it to the frame memory 74. On the other hand, if the switching unit 71 determines that the input from the image obtaining unit 52 is not the special light image (the normal light image), it outputs it to the motion vector detection unit 72 and the synthesis unit 77.

[0088] In step S2, the frame memory 74 supplies the special light image held until then to the feature extraction filter processing unit 75 and the motion correction unit 76. Also, the frame memory 74 updates the held special light image in a case where the special light image is input from the switching unit 71.

[0089] In step S3, the feature extraction filter processing unit 75 performs predetermined feature extraction filter processing on the special light image supplied from the frame memory 74, and thereby generates the extracted image of the fluorescent thread by extracting the area of the fluorescent thread 32 in the image, and outputs it to the motion correction unit 76.

[0090] In step S4, the motion vector detection unit 72 detects the motion vector using two normal light images at different imaging times and outputs it to the correction amount estimation unit 73.

[0091] In step S5, the correction amount estimation unit 73 determines whether the detected motion vector is less than or equal to a predetermined threshold. If so, the procedure proceeds to step S6 to use the motion vector for motion correction. Conversely, if the detected motion vector is greater than the predetermined threshold, it is not used for motion correction. In this case, the image synthesis processing corresponding to this imaging time ends.

[0092] In step S6, the correction amount estimation unit 73 estimates the motion correction amounts of the special light image and the extracted fluorescent thread image based on the motion vector detected by the motion vector detection unit 72, and outputs the estimated motion correction amounts to the motion correction unit 76. Specifically, the motion correction amounts H x and H y for example, as represented by the following equation (1): [Equation 1] Hx=∑t=1NVx, t Hy=∑t=1NVy, t

[0093] In equation (1) V x and V y the detected and corrected motion vectors and N represents the imaging time t = N of the normal light image in which the motion vector is detected, with respect to the imaging time t = 0 of the special light image to be corrected.

[0094] At this time, the correction amount estimation unit 73 may also estimate the motion correction amount after correcting a variation of the motion vectors based on a series of motion vectors as described below, as another method for estimating the motion correction amount.

[0095] Fig. 8 shows a processing flow of estimating the motion correction amount after correcting the variation of the motion vectors based on a series of motion vectors. Fig. Figure 9 shows an image of the correction of variation in motion vectors based on a series of motion vectors.

[0096] Specifically, the correction amount estimation unit 73 estimates the motion vector (V' x,t 'V' y,t ) for the imaging time t, as represented by the following equation (2): [Equation 2] V'x, t=axt3+bxt2+cxt+dx V'y, t=ayt3+byt2+cyt+dy

[0097] Then, the correction amount estimation unit 73 replaces the motion vector from equation (1) with the estimated motion vector (V' x,t 'V' y,t ) and calculates the movement correction amounts H x and H y by the following equation (3): [Equation 3] Hx=∑t=1NV'x, t Hy=∑t=1NV'y, t

[0098] The coefficients (a x , b x , c x , d x ) and (a y , b y , c y , d y ) in equation (2) by the least squares method using the detected motion vectors (V x,1 , V y,1 ), ..., and (V x,t , V y,t ) can be calculated.

[0099] After the movement correction amount is estimated as described above, the procedure goes to step S7.

[0100] In step S7, based on the motion correction amount input from the motion correction amount estimation unit 73, the motion correction unit 76 performs the motion correction of the special light image from the frame memory 74 and the motion correction of the extracted fluorescent filament image from the feature extraction filter processing unit 75, and outputs the motion-corrected special light image and the extracted fluorescent filament image to the synthesis unit 77. In the normal light image, the special light image, and the extracted fluorescent filament image input to the synthesis unit 77, objects in the respective images are accurately aligned.

[0101] In step S8, the synthesis unit 77 performs synthesis processing on the normal light image and the motion-corrected extracted fluorescent filament image, and outputs the resulting synthetic image to the image output unit 54. For example, briefly note that the synthesis unit 77 extracts pixels whose luminance or chrominance is greater than or equal to a predetermined threshold in the extracted fluorescent filament image as the fluorescent filament region 32. Then, the synthesis unit 77 generates the synthetic image by using the pixel value of the extracted fluorescent filament image for the extracted region of the fluorescent filament 32 and the pixel value of the normal light image for the other region, and generates the synthetic image therefrom.

[0102] In addition, the synthesis unit 77 can also perform, for example, the following synthesis processing. [Equation 4] OR(x, y)=C0×NR(x, y)+C1×featureR(x, y)×IR(x, y) OG(x, y)=C0×NG(x, y)+C1×featureG(x, y)×IG(x, y) OB(x, y)=C0×NB(x, y)+C1×FeatureB(x, y)×IB(x, y)

[0103] In Equation (4), O(x, y) represents the pixel value of the synthetic image, N(x, y) represents the pixel value of the normal light image, Feature(x, y) represents the pixel value of the motion-corrected extracted fluorescent filament image, and I(x, y) represents the pixel value of the motion-corrected special light image. C0 and C1 are coefficients for controlling the degree of superposition and can be arbitrarily set by the user. In the synthesis processing according to Equation (4), the result of multiplying the motion-corrected extracted fluorescent filament image and the special light image is added to the normal light image.

[0104] The synthesis unit 77 can also perform, for example, the following synthesis processing. [Equation 5] (OR(x,y)OG(x,y)OB(x,y))=(C1R×FeatureR(x,y) C1G×FeatureG(x, y) C1B×FeatureB(x, y)C2R×FeatureR(x,y) C2G×FeatureG(x, y) C1B×FeatureB(x, y)C3R×FeatureR(x,y) C3G×FeatureG(x, y) C1B×FeatureB(x, y))(NR(x, y)NG(x, y)NB(x, y))

[0105] In Equation (5), O(x, y) represents the pixel value of the synthetic image, N(x, y) represents the pixel value of the normal light image, Feature(x, y) represents the pixel value of the motion-corrected extracted fluorescent filament image, and C represents a color conversion coefficient. In the synthesis processing according to Equation (5), pseudo-color conversion is performed on the normal light image by color matrix processing using the color conversion coefficient C multiplied by the pixel values of the extracted fluorescent filament image.

[0106] In the image synthesis processing described above, the motion correction of the special light image and the extracted fluorescent thread image can be accurately performed because the motion vector is detected using only the normal light image and the motion correction amount is estimated after correcting the detected motion vector. This allows the area of the fluorescent thread 32 in the special light image to be accurately aligned with respect to the normal light image, and the synthetic image with improved visibility can be generated. Because the generated synthetic image is displayed on the display device 11, the image in which both the affected area and the fluorescent thread 32 can be easily visually recognized can be presented to the surgeon.

[0107] Although an example has been described in which the light source device 13 is controlled so that one special light image is obtained for four normal light images as shown in Fig. 4, the ratio between the number of generated normal light images and special light images is not limited to this. For example, as shown in Fig. 10, the control unit 51 can perform such control that the normal light image and the special light image are alternately obtained. <2. Second Embodiment>

[0108] Next, a second embodiment will be described. In the following description, previously described parts will be omitted, and only various parts will be described.

[0109] According to the first embodiment described above, the imaging unit 41 includes the CMOS sensor having the general Bayer array and the like, the special light image obtained by the special light and the normal light image obtained by the normal light are generated by changing the illumination condition of the light source device 13, and the synthetic image is generated by using the special light image and the normal light image.

[0110] According to the second embodiment, as the image sensor of an imaging unit 41, a sensor provided with a pixel sensitive to light of a narrow wavelength band is used, as shown in Fig. 11 is shown.

[0111] The Fig. The image sensor shown in Figure 11 has a pixel array in which four pixels NB, G, R, and B are repeatedly arranged. According to the second embodiment, a fluorescent filament 32 emits light of a specific narrow wavelength band under white light. The NB pixel is sensitive to the narrow wavelength band light emitted by the fluorescent filament 32.

[0112] In a case where the imaging unit 41 is provided with such an image sensor as in Fig. 12, both a pixel value obtained from the R, G, and B pixels for generating an image in which an operator can easily visually recognize an affected area and a pixel value obtained from the NB pixel for generating an image in which the operator can easily visually recognize the fluorescent thread 32 can be obtained by one-time imaging.

[0113] Then, an image processing unit 53 performs demosaic processing by using the pixel values of the R, G, and B pixels, thereby generating an RGB demosaic image in which the affected area can be easily visually recognized. It also generates an interpolated NB image in which the fluorescent thread 32 can be easily visually recognized by performing interpolation processing in which the NB pixel value is interpolated so that it has the same resolution as the RGB demosaic image. Subsequently, the image processing unit 53 performs synthesis processing in which the RGB demosaic image and the interpolated NB image are synthesized to generate a synthetic image, as shown in FIG. Fig. 12 and outputs this to an image output unit 54. <3. Other configuration example of the fluorescent thread>

[0114] In each of the embodiments described above, the fluorescent filament 32 is described as a material that uniformly glows in the same color in a portion of an entire filament.

[0115] However, the fluorescent thread 32 may also be designed to fluoresce in a repeating pattern that is not present in a living body, such as in Fig. 13. The Fig. The fluorescent thread 32 shown in Figure 13 has a pattern in which a long, medium, and short combination pattern of a second fluorescent color different from a first fluorescent color is repeated at a predetermined interval between the first fluorescent colors. Here, long, medium, and short indicate length differences by which the second fluorescent color extends in the longitudinal direction of the fluorescent thread 32.

[0116] By designing the fluorescent thread 32 in this way to glow in the repeating pattern not present in the living body, the operator can easily obtain a sense of depth of the affected area and toward the distal end of the thread displayed as an image based on the density or length of fluorescent patterns.

[0117] Fig. 14 shows another example of the repeating pattern not present in the living body of the fluorescent thread 32.

[0118] A from Fig. 14 shows an example of the repeating pattern having a configuration in which the first fluorescent color and the second fluorescent color having the same length are alternately arranged.

[0119] B of Figure 14 shows a pattern in which three patterns, namely a first pattern long, long, long, a second pattern medium, medium, medium and a third pattern short, short, short in the second fluorescent color, are repeated in the first fluorescent color.

[0120] A is from Fig. 14 and B from Fig. 14 merely illustrates examples of repeating patterns that do not exist in the living body, and the number, length, type, and the like of the repeating pattern are not limited to those shown in this example. The repeating pattern that does not exist in the living body may be a pattern in which patterns of multiple fluorescent colors are arranged according to a predetermined rule.

[0121] C from Fig. Figure 14 shows an example of the fluorescent thread 32 in which the fluorescent colors cannot be clearly distinguished but change gradually. At C from Fig. 14, the colors can change gradually along the entire length of the fluorescent thread 32, or the gradation at C from Fig. 14 may repeat multiple times throughout the fluorescent filament 32. Such a pattern of the fluorescent filament 32 is also included in the repeating pattern, which does not exist in the living body. <4. Variation of the synthesis unit>

[0122] Fig. 15 shows a variation of a synthesis unit 77 of an image processing unit 53.

[0123] The synthesis unit 77 is provided with a color conversion processing unit 78.

[0124] The color conversion processing unit 78 performs signal processing to convert the color of a fluorescent thread 32 in a synthetic image obtained by synthesizing a normal light image and a special light image so that it has a complementary color relationship with a background color. Since the background of the fluorescent thread 32 in a surgical site image is usually an organ of the patient (living body) in orange and red, the color conversion processing unit 78 converts the color of the fluorescent thread 32 into, for example, blue and green colors so that it complements the color of the organ.

[0125] Fig. 16 shows an example of the intraoperative operation using a thread with a repeating pattern, as in A of Fig. 14, a synthetic image produced as a fluorescent thread 32 and an image after color conversion processing, wherein the color conversion processing unit 78 of the synthesis unit 77 performs the color conversion.

[0126] The image in the upper right part of Fig. 16 shows the image after color conversion processing, wherein a first fluorescent color (for example, yellow) and a second fluorescent color (for example, green) of the fluorescent thread 32 in the synthetic image are converted into a color corresponding to a first complementary color (for example, yellow-green) and a color corresponding to a second complementary color (for example, blue), respectively.

[0127] The image in the lower right part of Fig. 16 shows the image after the color conversion processing, wherein the first fluorescent color (for example, yellow) and the second fluorescent color (for example, green) of the fluorescent thread 32 in the synthetic image are converted into a color corresponding to the complementary color (for example, blue).

[0128] The visibility of the fluorescent thread 32 can be further improved by exchanging the color of the fluorescent thread 32 in the synthetic image in this way, taking the background color into account.

[0129] Also, the color conversion processing unit 78 can detect a position in the depth direction and the distal end of the fluorescent thread 32 by detecting the density or the length of the repeating pattern not present in a living body, as described with reference to the Fig. 13 and Fig. 14, and convert them into different colors for display on the front and back of the fluorescent thread 32 and convert them into a different color at the distal end of the fluorescent thread 32. This makes it easier for the surgeon to detect the depth and the distal end of the fluorescent thread 32.

[0130] Although the color conversion processing unit 78 is a part of the synthesis unit 77 in Fig. 15, it can also be provided in a subsequent stage of the synthesis unit 77. <5. Other application example of color conversion processing>

[0131] Another application example of the color conversion processing by the color conversion processing unit 78 will be described.

[0132] During suturing, it is important to determine anteroposterior positions of a needle tip and the thread (on the front and back), but detecting anteroposterior relationship between the needle tip and the thread only by parallax during a long operation may cause operator fatigue.

[0133] Therefore, in a case where the thread is the fluorescent thread 32 because it emits light, it is possible to determine the antero-posterior relationship between the needle tip and the thread by determining whether the reflection of the light from the fluorescent thread 32 is on the needle.

[0134] In particular, as in Fig. 17, in a case where a needle 33 is located below the fluorescent thread 32, in other words, in a case where the needle 33 is located on the back side of the fluorescent thread 32 as seen from the imaging unit 41, there is a reflection of the light coming from the fluorescent thread 32 at the needle 33. On the other hand, in a case where the needle 33 is located above the fluorescent thread 32, in other words, in a case where the needle 33 is located in front of the fluorescent thread 32 as seen from the imaging unit 41, there is no reflection of the light coming from the fluorescent thread 32 at the needle 33.

[0135] In a case where it is determined that there is a reflection of the light coming from the fluorescent thread 32 on the needle 33, the color conversion processing unit 78 performs the color conversion so that the color of the needle 33 in the synthetic image is a first color (for example, red), and in a case where it is determined that there is no reflection of the light coming from the fluorescent thread 32 on the needle 33, it performs the color conversion so that the color of the needle 33 in the synthetic image is a second color (for example, blue).

[0136] Fig. 18 is a flowchart of the color conversion processing for converting the color of the needle in the synthetic image according to the antero-posterior relationship between the needle tip and the thread.

[0137] First, in step S21, the color conversion processing unit 78 recognizes the needle 33 and the fluorescent thread 32 in the synthetic image and determines whether the distance between the distal end of the needle 33 (needle tip) and the fluorescent thread 32 is within a predetermined range (distance).

[0138] Although the method for detecting the needle 33 in the synthetic image is not particularly limited, it is assumed that the needle 33 in the synthetic image can be detected by some methods.

[0139] If it is determined in step S21 that the distance between the distal end of the needle 33 and the fluorescent thread 32 is not within the predetermined range, the procedure goes to step S22, and the color conversion processing unit 78 outputs the synthetic image as it is to the image output unit 54 without performing the color conversion of the needle 33 in the synthetic image.

[0140] On the other hand, if it is determined in step S21 that the distance between the distal end of the needle 33 and the fluorescent thread 32 is within the predetermined range, the procedure goes to step S23, and the color conversion processing unit 78 determines whether reflection of the light coming from the fluorescent thread 32 on the needle 33 occurs in the synthetic image.

[0141] If it is determined in step S23 that there is a reflection of the light coming from the fluorescent thread 32 on the needle 33, the procedure goes to step S24 and the color conversion processing unit 78 converts the color of the needle 33 in the synthetic image into the first color (for example, red) and outputs it.

[0142] On the other hand, if it is determined in step S23 that there is no reflection of the light coming from the fluorescent thread 32 on the needle 33, the procedure goes to step S25 and the color conversion processing unit 78 converts the color of the needle 33 in the synthetic image into the second color (for example, blue) and outputs it.

[0143] By performing the color conversion processing described above, the operator can easily grasp the antero-posterior relationship between the needle tip and the thread simply by distinguishing the color of the needle 33 in the synthetic image displayed on a display device 11. <6. Pixel addition processing in the fluorescent thread area>

[0144] In general, spontaneously light-emitting materials have low light emission intensity, and there is a concern that images with a poor signal-to-noise ratio may result due to sensor noise, etc. Therefore, if there is a slight change in motion in a region of the fluorescent filament 32, the synthesis unit 77 can add pixel values of multiple consecutive images, thereby generating a synthetic image with an increased signal-to-noise ratio and improved visibility.

[0145] Fig. Figure 19 shows a flowchart of pixel addition processing in the fluorescent filament region, in which the pixel value of the fluorescent filament region 32 is added in the case of a slight motion change. This processing starts, for example, when the normal light image and a motion-corrected extracted fluorescent filament image are input to the synthesis unit 77.

[0146] First, in step S41, the synthesis unit 77 obtains the extracted fluorescent filament image of a Q-th frame and the extracted fluorescent filament image of a (Q+1)-th frame. Here, the Q-th and (Q+1)-th frames indicate the order in which only the extracted fluorescent filament images generated from the special light images generated at an interval of multiple frames are counted, as shown in Fig. 4 and Fig. 10. Therefore, the synthesis unit 77 needs to hold the extracted fluorescent filament image of the immediately preceding Q-th frame in an internal memory and the like until the extracted fluorescent filament image of the (Q+1)-th frame is input from a motion correction unit 76.

[0147] In step S42, the synthesis unit 77 sets a predetermined pixel of the extracted fluorescent thread image of the (Q+1)-th frame as a pixel of interest.

[0148] In step S43, the synthesis unit 77 detects a motion vector of the pixel of interest using the extracted fluorescent filament image of the Q-th frame and the extracted fluorescent filament image of the (Q+1)-th frame.

[0149] In step S44, the synthesis unit 77 determines whether the detected motion vector of the pixel of interest is smaller than a predetermined threshold.

[0150] If it is determined in step S44 that the detected motion vector of the pixel of interest is smaller than the predetermined threshold, the procedure goes to step S45 and causes the synthesis unit 77 to make a value obtained by adding the pixel value of the pixel of interest of the (Q+1)-th frame to the pixel value of the pixel of the Q-th frame equal to the pixel value of the pixel of interest of the (Q+1)-th frame.

[0151] On the other hand, if it is determined in step S44 that the detected motion vector of the pixel of interest is greater than or equal to the predetermined threshold, the process in step S45 is skipped. In this case, the pixel value of the pixel of interest of the (Q+1)th frame is not changed (not added).

[0152] In step S46, the synthesis unit 77 determines whether all pixels of the extracted fluorescent thread image of the (Q+1)-th frame are set as pixels of interest.

[0153] If it is determined in step S46 that not all pixels of the extracted fluorescent filament image of the (Q+1)th frame are set as pixels of interest, the procedure returns to step S42, and the above-described steps S42 to 46 are repeated. That is, in the extracted fluorescent filament image of the (Q+1)th frame, a pixel that is not yet set as a pixel of interest is set as the next pixel of interest, the motion vector is detected, and the pixel value is added based on the detection result.

[0154] Then, if it is determined in step S46 that all pixels of the extracted fluorescent filament image of the (Q+1)th frame have been set as pixels of interest, the pixel addition processing in the fluorescent filament area ends.

[0155] This pixel addition processing in the fluorescent thread area can be performed, for example, between the process in step S7 and the process in step S8 of the image synthesis processing in Fig. 7. Further, in the process in step S8 of the image synthesis processing in this case, the synthesis processing is executed to synthesize the extracted image of the fluorescent filament to which the pixel value has been added and the normal light image for the area of the fluorescent filament 32 with a small movement, so that the synthetic image is generated.

[0156] Fig. 20 shows an example of the extracted image of the fluorescent thread after performing the pixel addition processing in the fluorescent thread area.

[0157] The extracted image of the fluorescent thread from Fig. 20, the luminance of a region with less movement from the fluorescent filament 32 region is high. By performing pixel addition processing on the fluorescent filament region, the synthetic image can be generated in which the signal-to-noise ratio of the fluorescent filament 32 region with less movement is increased and the visibility is improved.

[0158] Here, the explanation of the pixel addition processing in the fluorescent filament region described above can be applied not only to the image synthesis processing for generating the normal light image and the special light image using an image sensor having a general Bayer array to generate the synthetic image, but also to a case where an image sensor having pixels sensitive to light of a narrow wavelength band, as in Fig. 11 is used.

[0159] The series of processes described above can be executed by hardware or software. In a case where a series of processes is executed by software, a program constituting the software can be installed on a computer. Here, the computer includes a computer constructed with dedicated hardware, a general-purpose personal computer that can perform various functions through various installed programs, and the like.

[0160] Fig. Fig. 21 is a block diagram showing a configuration example of the hardware of the computer that executes the above-described series of processes by a program.

[0161] In a computer 100, a central processing unit (CPU) 101, a read-only memory (ROM) 102 and a random access memory (RAM) 103 are connected to each other by a bus 104.

[0162] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0163] The input unit 106 includes a keyboard, a mouse, a microphone, and the like. The output unit 107 includes a display, a speaker, and the like. The storage unit 108 includes a hard disk, a non-volatile memory, and the like. The communication unit 109 includes a network interface and the like. The drive 110 drives a removable medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0164] In the computer 100 configured as described above, the CPU 101 loads the program stored in the storage unit 108 into the RAM 103 through, for example, the input / output interface 105 and the bus 104 to execute it, so that the series of processes described above are executed.

[0165] The computer 100 may be a so-called cloud computer, which is connected, for example, via the Internet.

[0166] The program executed by the computer 100 may be a program whose processes are executed in the chronological order described in this patent specification, or a program whose processes are executed in parallel or at necessary times, for example when a call is made.

[0167] The embodiments of the present technology are not limited to the embodiments described above, and various modifications can be made without departing from the scope of the present technology.

[0168] The image synthesis processing to which the present technology is applied is similarly applicable to a surgical system in which a video microscope is used in addition to the endoscopic surgical system described above. In this case, a synthetic image can be generated using the special light image and the normal light image generated by an imaging unit 41 of the video microscope.

[0169] In the embodiments described above, the term "frame period" is used as the interval of image formation by the imaging unit 41, but the interval at which the imaging unit 41 forms the imaged image (the number of images formed per second) can be appropriately set.

[0170] All or part of the several embodiments and variations thereof described above may be combined as needed.

[0171] The present technology can be interpreted as cloud computing, where a function is shared by multiple devices over the network to process it jointly.

[0172] Also, each step described in the flowchart above may be performed by one device or shared by multiple devices.

[0173] Furthermore, if multiple processes are included in one step, multiple processes included in one step may be executed by one device or divided by multiple devices.

[0174] The effects described in this patent specification are for illustrative purposes only and are not restrictive, and effects other than those described in this patent specification may also be included.

[0175] In this specification, a system is intended to mean an assembly of multiple components (devices, modules (parts), and the like), regardless of whether all components are housed in the same housing. Therefore, these systems mean multiple devices housed in different housings connected via the network and a device obtained by housing multiple modules in one housing.

[0176] The present technology can also have the following configurations. (1) An image processing apparatus comprising: an image obtaining unit that obtains a first image generated under an illumination condition in which a surgical thread fluoresces and a second image generated under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread, and a synthesis unit that generates a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image. (2) The image processing apparatus according to (1) as described above, wherein the first image is an image formed under an illumination condition in which light of a narrow wavelength band different from visible light is irradiated. (3) The image processing apparatus according to (1) as described above, wherein the first image is generated using a pixel signal of a narrowband pixel of signals imaged by an image sensor having an RGB pixel receiving light of an R, G, or B wavelength and the narrowband pixel receiving light of a narrow wavelength band other than visible light, and wherein the second image is generated using a pixel signal of the RGB pixel. (4) The image processing apparatus according to any one of (1) to (3) as described above, where the surgical thread fluoresces in a pattern that is not present in a living body. (5) The image processing apparatus according to (4) as described above, wherein in the pattern a second fluorescent color different from the first fluorescent color is repeated between the first fluorescent color. (6) The image processing apparatus according to (4) as described above, where the pattern is a gradation pattern of fluorescent colors. (7) The image processing apparatus according to any one of (1) to (6) as described above, further comprising: a color conversion processing unit that converts a color of the surgical thread in the synthetic image into a predetermined color. (8) The image processing apparatus according to (7) as described above, wherein the color conversion processing unit performs color conversion so that a complementary color relationship with a background color is realized. (9) The image processing apparatus according to (7) as described above, wherein the surgical thread fluoresces in a pattern that is not present in a living body, and the color conversion processing unit converts respective colors of the pattern into other colors. (10) The image processing apparatus according to any one of (1) to (9) as described above, further comprising: a color conversion processing unit that converts a color of a needle of the synthetic image according to an antero-posterior relationship between the surgical thread and the needle. (11) The image processing apparatus according to any one of (1) to (10) as described above, wherein the synthesis unit generates the first image by adding pixel values of a plurality of first images for a region of the surgical thread with less movement, and the synthetic image is generated, which is obtained by synthesizing the first image to which the pixel values have been added and the second image. (12) Image processing method comprising the following steps: Obtaining a first image produced under an illumination condition in which a surgical thread fluoresces and a second image produced under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread, and Generating a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image. (13) Operating system comprising: an imaging unit that generates a first image generated under an illumination condition in which a surgical thread fluoresces and a second image generated under an illumination condition including at least visible light as images of a surgical site using the fluorescent surgical thread, and a synthesis unit that generates a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image. (14) The operating system according to (13) as described above, further comprising: a light source device that emits light of a narrow wavelength band other than visible light if the first image is formed, and emits light including at least the visible light if the second image is formed. (15) Operating system according to (13), as described above, wherein the imaging unit comprises an image sensor having an RGB pixel that receives light of an R, G, or B wavelength, and a narrowband pixel that receives light of a narrow wavelength band other than visible light. (16) Surgical thread that fluoresces in a pattern not present in a living body. (17) Surgical thread according to (16), as described above, wherein in the pattern a second fluorescent color different from the first fluorescent color is repeated between the first fluorescent color. (18) Surgical thread according to (16), as described above, where the pattern is a gradation pattern of fluorescent colors. LIST OF REFERENCE SYMBOLS 10 Endoscopic surgical system 11 Display device 13 Light source device 19 Endoscope 32 fluorescent thread 33 needle 41 Imaging Unit 51 Control unit 52 image-preserving unit 53 Image processing unit 54 Image output unit 72 Motion vector detection unit 75 Feature extraction filter processing unit 77 Synthesis unit 78 Color conversion processing unit 100 computers 101 CPU 102 ROM 103 RAM 106 Input unit 107 Output unit 108 storage unit 109 Communication unit 110 drive

Claims

[1] Image processing device (53) comprising: an image obtaining unit (52) which obtains a first image generated under an illumination condition in which a surgical thread (32) fluoresces and a second image generated under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread (32), a synthesis unit (77) that generates a synthetic image obtained by synthesizing a portion of the surgical thread (32) of the first image and the second image; and a color conversion processing unit (78) which converts a color of a needle (33) of the synthetic image according to an antero-posterior relationship between the surgical thread (32) and the needle (33). [2] The image processing apparatus according to claim 1, wherein the first image is an image formed under an illumination condition in which light of a narrow wavelength band other than visible light is irradiated. [3] Image processing device (53) according to claim 1 or 2, wherein the first image is generated using a pixel signal of a narrowband pixel of signals imaged by an image sensor having an RGB pixel receiving light of an R, G, or B wavelength and the narrowband pixel receiving light of a narrow wavelength band other than visible light, and wherein the second image is generated using a pixel signal of the RGB pixel. [4] An image processing apparatus (53) according to any preceding claim, wherein the surgical thread (32) fluoresces in a pattern not present in a living body. [5] The image processing apparatus (53) according to claim 4, wherein in the pattern, a second fluorescent color different from the first fluorescent color is repeated between the first fluorescent color. [6] An image processing apparatus (53) according to claim 4 or 5, wherein the pattern is a gradation pattern of fluorescent colors. [7] Image processing device (53) according to one of the preceding claims, further comprising: a color conversion processing unit (78) that converts a color of the surgical thread (32) in the synthetic image into a predetermined color. [8] The image processing apparatus (53) according to claim 7, wherein the color conversion processing unit (78) performs color conversion so that a complementary color relationship with a background color is realized. [9] Image processing device (53) according to claim 7 or 8, wherein the surgical thread (32) fluoresces in a pattern that is not present in a living body, and the color conversion processing unit (78) converts respective colors of the pattern into other colors. [10] Image processing device (53) according to one of the preceding claims, wherein the synthesis unit (77) generates the first image by adding pixel values of a plurality of first images for a region of the surgical thread with less movement, and the synthetic image is generated, which is obtained by synthesizing the first image to which the pixel values have been added and the second image. [11] Image processing method comprising the following steps: Obtaining a first image produced under an illumination condition in which a surgical thread (32) fluoresces and a second image produced under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread; Generating a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image; and Converting a color of a needle (33) of the synthetic image according to an antero-posterior relationship between the surgical thread (32) and the needle (33). [12] Operating system (10) comprising: an imaging unit (41) that generates a first image generated under an illumination condition in which a surgical thread (32) fluoresces and a second image generated under an illumination condition comprising at least visible light as images of a surgical site using the fluorescent surgical thread (32), and a synthesis unit (77) that generates a synthetic image obtained by synthesizing a portion of the surgical thread of the first image and the second image; and a color conversion processing unit (78) configured to convert a color of a needle (33) of the synthetic image according to an antero-posterior relationship between the surgical thread (32) and the needle (33). [13] The operating system (10) of claim 12, further comprising: a light source device (13) which emits light of a narrow wavelength band other than visible light if the first image is formed, and emits light comprising at least the visible light if the second image is formed. [14] The operating system (10) according to claim 12 or 13, wherein the imaging unit (41) comprises an image sensor having an RGB pixel that receives light of an R, G or B wavelength and a narrow-band pixel that receives light of a narrow wavelength band other than visible light.

Images