Image processing device, method for operating image processing device, and program for image processing device

Abstract

[Problem] To provide an image processing device 1 that performs appropriate light source control. [Solution] The image processing device 1 is provided with: an FPGA 13 that performs a first processing on an image P0 from an endoscope 9 and outputs a primary image P1; and a GPU 12 that performs a second processing on the primary image P1 and outputs a secondary image P2. When the GPU 12 is normal, a detection processing for controlling the light amount of a light source device 8 is performed on the basis of the secondary image P2. When the GPU 12 is abnormal, the detection processing is performed on the basis of the primary image P1.

Description

Image processing device, operation method of image processing device, and program for image processing device

[0001] The present invention relates to an image processing device for processing medical images, a method for operating an image processing device for processing medical images, and a program for an image processing device for processing medical images.

[0002] In recent years, software processing using a GPU (Graphics Processing Unit) has come to be used in addition to hardware processing using an FPGA (Field Programmable Gate Array) for processing images acquired by medical equipment.

[0003] Japanese Patent Application Publication No. 2020-54834 discloses a signal processing device in which a primary image output by an FPGA is processed by a GPU to output a secondary image, but if the GPU is in an abnormal state, the signal processing device outputs the primary image.

[0004] Japanese Patent Application Laid-Open No. 2020-54834

[0005] For example, an image processing device for an endoscope processes an image of a subject illuminated by light from a light source device. For this reason, the image processing device not only generates an image to be displayed on a monitor, but also controls the light source device to adjust the brightness of the image.

[0006] In an image processing device in which a GPU processes a primary image output by an FPGA, if the GPU goes into an abnormal state or if GPU processing is omitted, there is a risk that the light source device will suddenly increase in light intensity or turn off.

[0007] An object of the embodiments of the present invention is to provide an image processing device that performs appropriate light source control, an operating method of an image processing device that performs appropriate light source control, and a program for an image processing device that performs appropriate light source control.

[0008] The image processing device of the embodiment comprises a hardware processing unit that performs a first processing on an image from a medical device and outputs a primary image, and a software processing unit that performs a second processing on the primary image and outputs a secondary image. When the software processing unit is normal, a detection processing for controlling the light intensity of the light source device is performed based on the secondary image, and when the software processing unit is abnormal, the detection processing is performed based on the primary image.

[0009] In the operating method of the image processing device of the embodiment, the hardware processing unit performs a first processing on an image from a medical device and outputs a primary image, the software processing unit performs a second processing on the primary image and outputs a secondary image, and if the software processing unit is normal, a detection processing for controlling the light intensity of the light source device is performed based on the secondary image, and if the software processing unit is abnormal, the detection processing is performed based on the primary image.

[0010] The program of the image processing device of the embodiment causes a computer to execute a process in which a hardware processing unit performs a first processing on an image from a medical device and outputs a primary image, a software processing unit performs a second processing on the primary image and outputs a secondary image, and if the software processing unit is normal, a detection processing is performed based on the secondary image to control the light intensity of the light source device, and if the software processing unit is abnormal, the detection processing is performed based on the primary image.

[0011] According to the embodiments of the present invention, it is possible to provide an image processing device that performs appropriate light source control, an operating method of an image processing device that performs appropriate light source control, and a program for an image processing device that performs appropriate light source control.

[0012] Fig. 1 is a perspective view of an endoscope system including an image processing device according to an embodiment. Fig. 2 is a configuration diagram of an endoscope system including an image processing device according to an embodiment. Fig. 3 is a flowchart of an operation method of the image processing device according to an embodiment. Fig. 4 is a flowchart of an operation method of an image processing device according to a modified example.

[0013] 1, an image processing device 1 according to this embodiment, together with an endoscope 9, a light source device 8, and a monitor 7, constitutes an endoscope system 2. The drawings based on the embodiment are schematic, and some components are not shown.

[0014] The endoscope 9 includes an insertion section 9A, a grip section 9E extending from the proximal end of the insertion section 9A, and a universal cord 9F extending from the grip section 9E. The insertion section 9A includes a distal end section 9B, a bending section 9C extending from the distal end section 9B for changing the direction of the distal end section 9B, and a flexible section 9D extending from the bending section 9C. An imager (not shown) including an optical system and an imaging element is disposed in the distal end section 9B. The universal cord 9F is connected to a light source device 8 and an image processing device 1. Illumination light generated by the light source device 8 passes through a light guide (not shown) passing through the universal cord 9F and the insertion section 9A, and is emitted from the distal end section 9B to illuminate the subject. The endoscope 9 is a so-called flexible endoscope, but a rigid endoscope may also be used.

[0015] The light source device 8 has a light source that generates illumination light to be supplied to the endoscope 9. The light source is a lamp such as a white LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, or a laser light source. The light source may also include a plurality of LEDs that each emit light of a different color.

[0016] The image processing device 1 controls the entire endoscope system 2, processes images from the endoscope 9, and generates images to be displayed on the monitor 7. Furthermore, the image processing device 1 performs detection processing to obtain the brightness of the image, and outputs a dimming signal to the light source device 8 to control the amount of light. The storage device 6 stores programs for the image processing device 1, etc.

[0017] As shown in FIG. 2 , the image processing device 1 has a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 12, an FPGA (Field Programmable Gate Array) 13, and a memory 14. The CPU 11 is a software processing unit that controls the entire image processing device 1. The GPU 12 is a software processing unit that performs image processing. The FPGA 13 is a hardware processing unit that performs image processing. The FPGA 13 can configure a logic circuit required by the user by combining programmable logic blocks. The memory 14 is a temporary storage unit such as a RAM.

[0018] In the present invention, the hardware processing unit (hardware processing circuit, logic circuit device) is not limited to the FPGA 13, but may be an ASIC (Application Specific Integrated Circuit), etc., as long as it executes processing using a hardware circuit such as a logic circuit. The software processing unit (software processing circuit, processor) is not limited to the GPU 12, but may be a CPU similar to the CPU 11, etc., as long as it executes processing using a program.

[0019] Compared to hardware processing units, software processing units are easier to change the processing content of and are less expensive, but software processing units can go into an abnormal state due to a lack of resources such as memory, overheating of the processor, or a bug in the program.

[0020] As will be described later, the image processing device 1 includes a hardware processing unit that performs a first processing on an image P0 from a medical device and outputs a primary image P1, and a software processing unit that performs a second processing on the primary image P1 and outputs a secondary image P2. When the software processing unit is normal, detection processing for controlling the light intensity of the light source device 8 is performed based on the secondary image P2, and when the software processing unit is abnormal, the detection processing is performed based on the primary image P1.

[0021] Therefore, the image processing device 1 can always perform appropriate light source control.

[0022] <Operation Method of Image Processing Apparatus> The operation method of the image processing apparatus 1 will be described with reference to the flowchart of FIG.

[0023] <Step S10> Image Input An image captured by the endoscope 9, for example, a single frame image of a moving image, is input to the image processing device 1. Note that the image processing device of the embodiment may also process an image from a medical device that outputs an image of a subject illuminated by light from a light source device, for example, a medical microscope.

[0024] <Step S20> First Processing A first processing is performed to convert the RAW image P0 from the endoscope 9 into a primary image P1 that can be displayed on the monitor 7. The FPGA 13, which is a hardware processing unit, performs the first processing.

[0025] For example, the first processing includes one of defective pixel correction, sensitivity correction, and shading correction.

[0026] <Step S30> GPU normal? If the GPU 12 is normal (YES), the GPU 12, which is a software processing unit, performs the second process. If the GPU 12 is not normal (NO), the second process is not performed and the process proceeds to step S50.

[0027] An abnormality in the GPU 12 is detected, for example, by having the GPU 12 perform a predetermined calculation. An example of the predetermined calculation is "n = n + 1" (n is an integer). When the calculation result is correct, the GPU 12 is determined to be normal. When the calculation result is incorrect or when no calculation result can be obtained, the GPU 12 is determined to be abnormal. The temperature of the GPU 12 may be monitored, and if it exceeds a predetermined temperature, the GPU 12 may be determined to be abnormal. The abnormality detection process for the GPU 12 is performed, for example, by the FPGA 13 or the CPU 11.

[0028] <Step S40> Second Processing The GPU 12, which is a software processing unit, performs second processing on the primary image P1 and outputs a secondary image P2.

[0029] For example, the secondary processing includes any one of AGC, gamma correction, magnification chromatic aberration correction, and white balance processing.

[0030] The detection of an abnormality in the GPU 12 may be performed by comparing the images (primary image P1, secondary image P2) before and after the second process. That is, step S30 may be performed after step S40. For example, if the volume of the secondary image P2 is significantly smaller than the volume of the primary image P1, or if the volume of the primary image P1 and the volume of the secondary image P2 are exactly the same, the GPU 12 is determined to be abnormal.

[0031] In the above case, an abnormality in the GPU 12 cannot be detected until step S40 (second processing) is completed. Therefore, the primary image P1 output by the FPGA 13 is output to the GPU 12 and also temporarily stored in the memory 14. Then, when the second processing is completed and it is detected that the GPU 12 is not normal, the primary image P1 is read from the memory 14, and detection processing (S50) is performed based on the primary image P1.

[0032] <Step S50> Detection Processing In order to adjust the light intensity of the light source device 8, detection processing is performed based on the primary image P1 or the secondary image P2. For example, the effective image area is divided into nine (3 x 3) areas, and the average luminance value, maximum luminance value, and number of pixels with luminance equal to or greater than a predetermined threshold value are calculated for each area. The detection processing is performed by a component of the image processing device 1 other than the GPU 12, which is a software processing unit, such as the FPGA 13, which is a hardware processing unit, or the CPU, which is a software processing unit, or the light source device 8.

[0033] Furthermore, the primary image P1 or the secondary image P2 is output to a monitor 7.

[0034] In addition, when an abnormality in the GPU 12 is detected, it is preferable that the FPGA 13, which is a hardware processing unit, performs simple image processing on the primary image P1, such as optical black correction processing, white balance processing, gamma correction processing, or resizing processing, to generate a tertiary image, and output the tertiary image to the monitor 7.

[0035] In the image processing device 1, if the GPU 12 is normal (S30; YES), detection processing is performed based on the secondary image P2, and if the GPU 12 is abnormal (S30; NO), detection processing is performed based on the primary image P1.

[0036] It should be noted that the detection processes may be different in the first detection process performed based on the secondary image P2 and the second detection process performed based on the primary image P1. The first detection process, for example, divides the effective image area into nine (3 x 3) areas, and calculates the average brightness, maximum brightness, and number of pixels with brightness values ​​equal to or greater than a predetermined threshold for each area. The second detection process, for example, calculates the average brightness over the entire effective image area, calculates the number of saturated pixels, and reduces the effective image area to (1 / 4 size) before calculating integrated data.

[0037] <Step S60> Generate Light Control Signal Based on the result of the brightness detection process, the CPU 11, for example, generates a light control signal for adjusting the brightness of the image.

[0038] <Step S70> Light Source Control Based on the dimming signal, the light amount of the light source device 8 is controlled. That is, the light amount is adjusted so that the brightness of the image becomes appropriate.

[0039] As explained above, the operating method of the image processing device 1 of this embodiment is such that the hardware processing unit (FPEG 13) performs a first processing on an image from the medical device (endoscope 9) and outputs a primary image, the software processing unit (GPU 12) performs a second processing on the primary image and outputs a secondary image, and if the software processing unit (GPU 12) is normal, detection processing for controlling the light intensity of the light source device 8 is performed based on the secondary image, and if the software processing unit (GPU 12) is abnormal, the detection processing is performed based on the primary image.

[0040] According to the operating method of the image processing device 1 of this embodiment, appropriate light source control is always performed.

[0041] <Modification> An image processing device 1A of the modification is similar to the image processing device 1 of the embodiment and has the same effects as the image processing device 1. For this reason, components with the same functions as those of the image processing device 1 are given the same reference numerals and descriptions thereof will be omitted.

[0042] There is a time lag between the detection of an abnormality in the GPU 12 and the generation of a dimming signal, during which the maximum amount of light may be emitted, or conversely, the amount of light may be significantly reduced, causing the monitor 7 to go into a blackout state.

[0043] In the image processing device 1A, if the GPU 12 is abnormal, the FPEG 13 controls the light amount of the light source device 8 to be fixed at a predetermined value until the detection process based on the primary image P1 is completed.

[0044] The operation method of the image processing apparatus 1A will be described with reference to the flowchart of Fig. 4. Note that the description of the same processes (steps) as those in the flowchart of Fig. 3 will be omitted.

[0045] <Step S30> GPU normal? If the GPU 12 is not normal, the second process (S40) is skipped, and the detection process (S50) using the primary image is started. At the same time, the process from step S62 is performed.

[0046] <Step S62> Acquire provisional dimming signal A provisional dimming signal for controlling the light source of the light source device 8 to a predetermined value is acquired. The provisional dimming signal is stored in advance, for example, in the memory 14. The predetermined value of the light amount is, for example, 50% of the maximum light amount generated by the light source device 8. The provisional dimming signal may be a dimming signal acquired by detection processing based on the secondary image P2 immediately before the occurrence of the abnormality.

[0047] <Step S64> Light Source Control The provisional dimming signal is sent to the light source device 8, and the light source device 8 is controlled to a predetermined light amount.

[0048] <Step S66> Is light modulation signal generation completed? When the processes from step S50 to step S60 are completed and the light modulation signal obtained by the detection process based on the primary image P1 is obtained, the processes from step S70 are performed.

[0049] The image processing device 1A can always perform appropriate light source control even if a time lag occurs between when an abnormality in the GPU 12 is detected and when a dimming signal is generated.

[0050] The program of the image processing device of the embodiment causes a computer to execute a process in which a hardware processing unit of the image processing device performs a first processing on an image from a medical device and outputs a primary image, a software processing unit of the image processing device performs a second processing on the primary image and outputs a secondary image, and if the software processing unit is normal, a detection processing is performed based on the secondary image to control the light intensity of the light source device, and if the software processing unit is abnormal, the detection processing is performed based on the primary image.

[0051] The program for causing the image processing device 1 to execute the operations described above is stored in whole or in part on a non-transitory computer-readable storage medium such as a flexible disk, a CD-ROM, or a hard disk.

[0052] The program may be distributed or provided in whole or in part via a communications network. A user may download the program via a communications network and install it on a computer, or install it on a computer from a storage medium, thereby enabling the operation of the image processing device of the present invention.

[0053] Furthermore, at least some of the components of the image processing apparatus of the present invention may be components of a server connected via a network.

[0054] The steps in the flowcharts in this specification may be executed in a different order, may be executed simultaneously, or may be executed in a different order, as long as this does not contradict the nature of the steps.

[0055] The ranges of the numerical values ​​described above are not limited to the ranges described above and can be increased or decreased as appropriate. Furthermore, the present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the scope of the present invention.

[0056] REFERENCE SIGNS LIST 1, 1A... Image processing device 2... Endoscope system 6... Storage device 7... Monitor 8... Light source device 9... Endoscope 11... CPU 12... GPU 13... FPGA 14... Memory

Claims

1. An image processing device comprising: a hardware processing unit that performs a first processing on an image from a medical device and outputs a primary image; and a software processing unit that performs a second processing on the primary image and outputs a secondary image, wherein when the software processing unit is normal, detection processing is performed based on the secondary image to control the amount of light from a light source device, and when the software processing unit is abnormal, the detection processing is performed based on the primary image.

2. The image processing device according to claim 1, wherein the hardware processing unit is an FPGA, and the software processing unit is a GPU.

3. An image processing device as described in claim 1, characterized in that if the software processing unit is abnormal, the light intensity of the light source device is fixed to a predetermined value until the detection processing based on the primary image is completed.

4. The image processing device according to claim 3, characterized in that the predetermined value of the light amount is a value obtained by the software processing unit through the detection process based on the secondary image immediately before the occurrence of an abnormality.

5. An image processing device according to claim 1, characterized in that the detection processing is performed using different detection methods for the first detection processing based on the secondary image and the second detection processing based on the primary image.

6. The image processing device according to claim 1, characterized in that the first processing includes one of defective pixel correction, sensitivity correction, and shading correction, and the second processing includes one of AGC, gamma correction, magnification chromatic aberration correction, and white balance processing.

7. The image processing device according to claim 1, wherein the abnormality in the software processing unit is detected based on the result of a predetermined calculation performed by the software processing unit.

8. The image processing device according to claim 1, wherein the abnormality in the software processing unit is detected based on a comparison process between the primary image and the secondary image.

9. The image processing device according to claim 1, wherein the detection processing is performed by the hardware processing unit.

10. The image processing device according to claim 1, wherein the medical device is an endoscope.

11. A method for operating an image processing device, characterized in that a hardware processing unit performs a first processing on an image from a medical device and outputs a primary image, a software processing unit performs a second processing on the primary image and outputs a secondary image, and when the software processing unit is normal, a detection processing for controlling the light intensity of a light source device is performed based on the secondary image, and when the software processing unit is abnormal, the detection processing is performed based on the primary image.

12. A program for an image processing device, characterized in that the program causes a computer to execute the following processing: a hardware processing unit of an image processing device performs a first processing on an image from a medical device and outputs a primary image; a software processing unit of the image processing device performs a second processing on the primary image and outputs a secondary image; and if the software processing unit is normal, a detection processing is performed based on the secondary image to control the amount of light from a light source device; and if the software processing unit is abnormal, the detection processing is performed based on the primary image.

Images