Image processing apparatus, operating method of image processing apparatus, and program of image processing apparatus
By combining hardware and software processing units, the image processing device can still perform appropriate light source control when the GPU malfunctions, solving the problem of unstable light source control caused by GPU malfunctions and achieving stability and reliability in image processing.
Patent Information
- Application Number
- CN202480090635.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-10
- Publication Date
- 2026-11-10
AI Technical Summary
In image processing devices, when the GPU malfunctions or skips GPU processing, the amount of light from the light source may increase or decrease dramatically, making it impossible to perform proper light source control.
The image processing device employs a combination of a hardware processing unit and a software processing unit. The hardware processing unit performs first processing on the image and outputs a single image. Under normal circumstances, the software processing unit performs second processing and controls the light source. In case of an anomaly, it performs detection processing based on the single image to ensure the stability of the light source control.
It enables appropriate light source control even when the GPU malfunctions, avoiding drastic changes in light intensity and ensuring the stability and reliability of image processing.
Smart Images

Figure CN123076340A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image processing apparatus for processing medical images, a method for operating the image processing apparatus for processing medical images, and a program for the image processing apparatus for processing medical images. Background Technology
[0002] In recent years, in the processing of images acquired by medical devices, in addition to hardware processing using FPGA (Field Programmable Gate Array), software processing using GPU (Graphics Processing Unit) has also been used.
[0003] Japanese Patent Application Publication No. 2020-54834 discloses a signal processing device in which a GPU processes a single image output from an FPGA to output a double image, but outputs a single image when the GPU is in an abnormal state.
[0004] Existing technical documents
[0005] Patent documents
[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-54834 Summary of the Invention
[0007] The problem that the invention aims to solve
[0008] For example, the image processing unit for endoscopes processes the image of a subject illuminated by light from a light source device. Therefore, the image processing unit not only generates the image displayed on the monitor, but also controls the light source device to adjust the brightness of the image.
[0009] In an image processing device where the GPU processes a single image output from the FPGA, when the GPU becomes abnormal or GPU processing is omitted, the light source may experience a sharp increase in light intensity or become extinguished.
[0010] The purpose of embodiments of the present invention is to provide an image processing apparatus for performing appropriate light source control, a method for operating the image processing apparatus for performing appropriate light source control, and a program for the image processing apparatus for performing appropriate light source control.
[0011] Methods for solving problems
[0012] The image processing apparatus of the embodiment includes: a hardware processing unit that performs first processing on an image from a medical device and outputs a single image; and a software processing unit that performs second processing on the single image and outputs a second image. When the software processing unit is functioning normally, it performs detection processing for controlling the amount of light from a light source device based on the second image. When the software processing unit malfunctions, it performs the detection processing based on the single image.
[0013] In the operation method of the image processing apparatus in the embodiment, the hardware processing unit performs first processing on the image from the medical device and outputs a single image, the software processing unit performs second processing on the single image and outputs a second image, and when the software processing unit is normal, detection processing for controlling the light amount of the light source device is performed based on the second image, and when the software processing unit is abnormal, the detection processing is performed based on the single image.
[0014] The program of the image processing apparatus of the embodiment causes the computer to perform the following processing: the hardware processing unit performs a first processing on an image from a medical device and outputs a single image; the software processing unit performs a second processing on the single image and outputs a second image; when the software processing unit is functioning normally, detection processing for controlling the amount of light from a light source device is performed based on the second image; when the software processing unit is malfunctioning, the detection processing is performed based on the single image.
[0015] Invention Effects
[0016] According to embodiments of the present invention, an image processing apparatus for performing appropriate light source control, a method for operating the image processing apparatus for performing appropriate light source control, and a program for the image processing apparatus for performing appropriate light source control can be provided. Attached Figure Description
[0017] Figure 1 This is a perspective view of an endoscope system including an image processing apparatus for implementing an embodiment.
[0018] Figure 2 This is a structural diagram of an endoscope system including an image processing apparatus for implementing an embodiment.
[0019] Figure 3 This is a flowchart illustrating the operation method of the image processing apparatus in the embodiment.
[0020] Figure 4 This is a flowchart of the working method of the modified image processing device. Detailed Implementation
[0021] <Endoscopic System>
[0022] like Figure 1As shown, the image processing apparatus 1 of this embodiment, together with the endoscope 9, the light source device 8, and the monitor 7, constitutes the endoscope system 2. The accompanying drawings based on this embodiment are schematic. Some of the constituent elements are omitted from the illustration.
[0023] The endoscope 9 includes an insertion portion 9A, a holding portion 9E extending from the base of the insertion portion 9A, and a universal cable 9F extending from the holding portion 9E. The insertion portion 9A includes: a front end portion 9B; a curved portion 9C extending from the front end portion 9B for changing the direction of the front end portion 9B; and a flexible portion 9D extending from the curved portion 9C. An imager (not shown) including an optical system and an imaging element is disposed at the front end portion 9B. The universal cable 9F is connected to a light source device 8 and an image processing device 1. Illumination light generated by the light source device 8 is emitted from the front end portion 9B via the universal cable 9F and a light guide (not shown) inserted into the insertion portion 9A to illuminate the subject. The endoscope 9 is a so-called flexible endoscope, but it can also be a rigid endoscope.
[0024] The light source device 8 has a light source that generates illumination light supplied to the endoscope 9. The light source is a white LED (Light Emitting Diode), xenon lamp, halogen lamp, or laser light source. Alternatively, the light source may include multiple LEDs that emit light of different colors.
[0025] The image processing device 1 controls the entire endoscope system 2 and processes the images from the endoscope 9 to generate an image displayed on the monitor 7. Furthermore, the image processing device 1 performs detection processing to obtain the brightness of the image and outputs a dimming signal for controlling the amount of light to the light source device 8. The storage device 6 stores the program of the image processing device 1, etc.
[0026] like Figure 2 As shown, the image processing device 1 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 12, an FPGA (Field Programmable Gate Array) 12, and a memory 14. The CPU 11 is the software processing unit that controls the entire image processing device 1. The GPU 12 is the software processing unit that performs image processing. The FPGA 13 is the hardware processing unit that performs image processing. The FPGA 13 can combine programmable logic blocks to construct the logic circuits required by the user. The memory 14 is a temporary storage unit such as RAM.
[0027] In this invention, the hardware processing unit (hardware processing circuit, logic circuit device) can perform processing through hardware circuits such as logic circuits, and is not limited to FPGA13, but can also be ASIC (Application Specific Integrated Circuit), etc. The software processing unit (software processing circuit, processor) can perform processing through programs, and is not limited to GPU12, but can also be a CPU, the same as CPU11, etc.
[0028] Compared to the hardware processing department, the software processing department is easier and less costly to change the content it processes. However, the software processing department may become abnormal due to insufficient resources such as memory, processor overheating, or program errors.
[0029] As described below, the image processing apparatus 1 includes: a hardware processing unit that performs first processing on an image P0 from a medical device and outputs a single image P1; and a software processing unit that performs second processing on the single image P1 and outputs a second image P2. When the software processing unit is functioning normally, detection processing for controlling the light intensity of the light source device 8 is performed based on the second image P2; when the software processing unit malfunctions, the detection processing is performed based on the single image P1.
[0030] Therefore, the image processing device 1 is always able to perform appropriate light source control.
[0031] <How an Image Processing Device Works>
[0032] according to Figure 3 The flowchart illustrates the operation of the image processing device 1.
[0033] <Step S10> Image Input
[0034] The image captured by the endoscope 9, such as a frame of a moving image, is input to the image processing device 1. Furthermore, the image processing device in this embodiment can also process images from a medical device, such as a medical microscope, which outputs an image of the subject illuminated by light from a light source device.
[0035] <Step S20> First Processing
[0036] A first process is performed to transform the RAW image P0 from the endoscope 9 into a primary image P1 that can be displayed on the monitor 7. The FPGA 13, which is the hardware processing unit, performs the first process.
[0037] For example, the first process includes any one of defective pixel correction, sensitivity correction, and shadow correction.
[0038] <Step S30> Is the GPU working properly?
[0039] If GPU 12 is functioning normally (Yes), GPU 12, as the software processing unit, performs the second processing. If GPU 12 is malfunctioning (No), the second processing is not performed, and the processing proceeds to step S50.
[0040] GPU12 anomalies are detected, for example, by having GPU12 perform a prescribed operation. The prescribed operation is, for example, "n = n + 1" (where n is an integer). If the operation result is correct, GPU12 is considered normal. If the operation result is incorrect or no result is obtained, GPU12 is considered abnormal. Alternatively, the temperature of GPU12 can be monitored; if it exceeds a specified temperature, GPU12 is considered abnormal. GPU12 anomaly detection and handling are performed, for example, by FPGA13 or CPU11.
[0041] <Step S40> Second Processing
[0042] GPU12, as the software processing unit, performs a second processing on the first image P1 and outputs the second image P2.
[0043] For example, the second processing includes any one of AGC, gamma correction, magnification color difference correction, and white balance processing.
[0044] Alternatively, anomaly detection of GPU12 can also be performed by comparing the images before and after the second processing (image P1 in step 1, image P2 in step 2). That is, step S30 can also be performed after step S40. For example, if the size of image P2 in step 2 is significantly reduced compared to the size of image P1 in step 1, or if the size of image P1 in step 1 is exactly the same as the size of image P2 in step 2, GPU12 is determined to be an anomaly.
[0045] In the above situation, if step S40 (second processing) is not completed, the abnormality of GPU 12 cannot be detected. Therefore, the first image P1 output by FPGA 13 is output to GPU 12 and temporarily stored in memory 14. Furthermore, when the second processing is detected to be completed and GPU 12 is malfunctioning, the first image P1 is read from memory 14, and detection processing based on the first image P1 is performed (S50).
[0046] <Step S50> Detection Processing
[0047] To adjust the light intensity of the light source device 8, detection processing is performed based on either the first image P1 or the second image P2. For example, the effective image area is divided into nine regions (3×3), and the average brightness, maximum brightness, and number of pixels with brightness above a specified threshold are calculated for each region. The detection processing is performed by components of the image processing device 1 other than the GPU 12 (which is a software processing unit), such as the FPGA 13 (which is a hardware processing unit), the CPU (which is a software processing unit), or the light source device 8.
[0048] In addition, either the first image P1 or the second image P2 is output to monitor 7.
[0049] In addition, if an anomaly is detected in GPU 12, FPGA 13, which is the hardware processing unit, preferably performs simple image processing on the first image P1, such as optical black correction processing, white balance processing, gamma correction processing, or size adjustment processing, to generate a third image, and outputs the third image to monitor 7.
[0050] In the image processing apparatus 1, when the GPU 12 is functioning normally (S30; Yes), detection processing is performed based on two images P2, and when the GPU 12 is malfunctioning (S30; No), detection processing is performed based on one image P1.
[0051] Furthermore, regarding the detection processing, the first detection processing based on two images P2 and the second detection processing based on one image P1 can be different detection methods. In the first detection method, for example, the effective image region is divided into nine regions (3×3), and for each region, the average brightness, the maximum brightness, and the number of pixels with a brightness value above a predetermined threshold are calculated. In the second detection method, for example, the average brightness across the entire effective image region is calculated, the number of saturated pixels is calculated, and cumulative data is calculated after the effective image region is reduced to (1 / 4 size).
[0052] <Step S60> Dimming signal generation
[0053] For example, the CPU11 generates a dimming signal for adjusting the brightness of the image based on the result of brightness detection processing.
[0054] <Step S70> Light Source Control
[0055] The amount of light from the light source device 8 is controlled based on the dimming signal. That is, the amount of light is adjusted in a way that makes the brightness of the image appropriate.
[0056] As explained above, in the operation method of the image processing apparatus 1 in this embodiment, the hardware processing unit (FPEG13) performs first processing on the image from the medical device (endoscope 9) and outputs a single image, and the software processing unit (GPU12) performs second processing on the single image and outputs a double image. When the software processing unit (GPU12) is functioning normally, detection processing for controlling the amount of light from the light source device 8 is performed based on the double image. When the software processing unit (GPU12) is malfunctioning, the detection processing is performed based on the single image.
[0057] According to the operating method of the image processing apparatus 1 of this embodiment, appropriate light source control is always performed.
[0058] <Variation Example>
[0059] The modified image processing apparatus 1A is similar to the image processing apparatus 1 of the embodiment and has the same effects as the image processing apparatus 1. Therefore, the same reference numerals are used to mark the components that have the same functions as the image processing apparatus 1, and the description is omitted.
[0060] There is a time lag between detecting the GPU12 anomaly and generating the dimming signal. During this time lag, it is possible for the maximum amount of illumination light to be emitted, or conversely, for the amount of light to be significantly reduced, causing the monitor 7 to become a black screen.
[0061] In the image processing apparatus 1A, in the event of a GPU 12 malfunction, FPEG13 controls the light source apparatus 8 to be fixed at a predetermined value until the detection processing based on the first image P1 is completed.
[0062] according to Figure 4 The flowchart illustrates the operation of the image processing apparatus 1A. Additionally, regarding... Figure 3 The same processing (steps) as the flowchart are omitted.
[0063] <Step S30> Is the GPU working properly?
[0064] If GPU12 malfunctions, the second processing step (S40) is omitted, and the first image detection processing (S50) is started. At the same time, processing from step S62 onwards is performed.
[0065] <Step S62> Obtain temporary dimming signal
[0066] A temporary dimming signal is obtained to control the light source of the light source device 8 to a predetermined value. The temporary dimming signal is, for example, pre-stored in the memory 14. The predetermined value of the light amount is, for example, 50% of the maximum light amount generated by the light source device 8. The temporary dimming signal may also be a dimming signal obtained by detection processing based on the two images P2 immediately before the abnormality is generated.
[0067] <Step S64> Light source control
[0068] A temporary dimming signal is sent to the light source device 8, and the light source device 8 is controlled to a specified amount of light.
[0069] <Step S66> Dimming signal generation complete?
[0070] When the processing of steps S50 to S60 is completed and the dimming signal obtained by the detection processing based on the first image P1 is obtained, the processing from step S70 onwards is performed.
[0071] The image processing device 1A can always perform appropriate light source control from the time it detects the anomaly of the GPU 12 until it generates the dimming signal, even if there is a time delay.
[0072] The program of the image processing apparatus of the embodiment causes the computer to perform the following processing: the hardware processing unit of the image processing apparatus performs a first processing on an image from a medical device and outputs a single image; the software processing unit of the image processing apparatus performs a second processing on the single image and outputs a second image; when the software processing unit is functioning normally, detection processing for controlling the amount of light from a light source device is performed based on the second image; when the software processing unit is malfunctioning, the detection processing is performed based on the single image.
[0073] The program used by the image processing device 1 to perform the actions described above is stored, in whole or in part, in a non-transitory computer-readable storage medium such as a floppy disk, CD-ROM, or hard disk.
[0074] The entire or a part of this program can be distributed or provided via a communication network. Users can download and install this program on a computer via a communication network, or install it on a computer from a storage medium, thereby enabling the image processing apparatus of the present invention to operate.
[0075] In addition, at least a portion of the components of the image processing apparatus of the present invention may also be components of a server connected via a network.
[0076] The execution order of each step in the flowchart in this manual can be changed, multiple steps can be executed simultaneously, or steps can be executed in different orders, as long as their nature is not violated.
[0077] The range of values described above is not limited to the range described above, and may be appropriately increased or decreased. Furthermore, the present invention is not limited to the embodiments described above, and various changes and modifications can be made without altering the spirit of the invention.
[0078] Explanation of reference numerals in the attached figures
[0079] 1, 1A…Image Processing Device
[0080] 2…Endoscopic System
[0081] 6… Storage devices
[0082] 7…monitor
[0083] 8…Light source device
[0084] 9…Endoscope
[0085] 11…CPU
[0086] 12…GPU
[0087] 13…FPGA
[0088] 14… Memory
Claims
1. An image processing apparatus, characterized in that, The image processing device has: The hardware processing unit performs initial processing on the image from the medical device and outputs a single image. as well as The software processing unit performs a second processing on the first image and outputs a second image. When the software processing unit is functioning normally, detection processing for controlling the amount of light from the light source device is performed based on the two images. When the software processing unit is malfunctioning, the detection processing is performed based on the first image.
2. The image processing apparatus according to claim 1, characterized in that, The hardware processing unit is an FPGA. The software processing unit is a GPU.
3. The image processing apparatus according to claim 1, characterized in that, In the event of an malfunction in the software processing unit, the amount of light from the light source device is fixed at a predetermined value until the detection processing based on the first image is completed.
4. The image processing apparatus according to claim 3, characterized in that, The specified value of the light quantity is obtained by the detection processing performed on the two images just before the anomaly occurs in the software processing unit.
5. The image processing apparatus according to claim 1, characterized in that, In the detection process, the first detection process based on the two images and the second detection process based on the one image are different detection methods.
6. The image processing apparatus according to claim 1, characterized in that, The first process includes any one of defect pixel correction, sensitivity correction, and shadow correction. The second processing includes any one of AGC, gamma correction, magnification color difference correction, and white balance processing.
7. The image processing apparatus according to claim 1, characterized in that, The anomaly in the software processing unit is detected based on the calculation results specified by the software processing unit.
8. The image processing apparatus according to claim 1, characterized in that, The anomaly in the software processing unit is detected by comparing the first image with the second image.
9. The image processing apparatus according to claim 1, characterized in that, The detection processing is performed by the hardware processing unit.
10. The image processing apparatus according to claim 1, characterized in that, The medical device is an endoscope.
11. A method of operating an image processing apparatus, characterized in that, The hardware processing unit performs initial processing on the image from the medical device and outputs a single image. The software processing unit performs a second processing on the first image and outputs a second image. When the software processing unit is functioning normally, detection processing for controlling the amount of light from the light source device is performed based on the two images. When the software processing unit is malfunctioning, the detection processing is performed based on the first image.
12. A program for an image processing apparatus, characterized in that, The computer will perform the following processing: The hardware processing unit of the image processing device performs initial processing on the image from the medical device and outputs a single image. The software processing unit of the image processing device performs a second processing on the first image and outputs a second image. When the software processing unit is functioning normally, detection processing for controlling the amount of light from the light source device is performed based on the two images. When the software processing unit is malfunctioning, the detection processing is performed based on the first image.
Citation Information
Patent Citations
Signal processing device and signal processing method
JP2020054834A
