Image processing device, image processing method, and recording medium

Abstract

This image processing device comprises a processor. The processor receives a far-point image in which a far point is in focus and a near-point image in which a near point is in focus (S1). The far-point image and the near-point image have different exposure amounts. Thereafter, the processor calculates the synthesis rate of each region of the far-point image and the near-point image on the basis of the brightness or frequency of each region of one of the far-point image and the near-point image and a predetermined relationship involving the brightness or frequency (S2), and synthesizes the far-point image and the near-point image using the calculated synthesis rate of each region (S3).

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a recording medium. [Background technology]

[0002] Conventionally, EDOF (Extended Depth of Field) Field Techniques for generating EDOF (extended depth of field) images and HDR (High Dynamic Range) images are known (see, for example, Patent Documents 1 and 2). An EDOF image is an image with an extended depth of field that is synthesized from multiple images with different focal lengths. An HDR image is an image with an extended dynamic range that is synthesized from multiple images with different exposures.

[0003] In Patent Documents 1 and 2, a method is proposed to extract an image from a plurality of images with different focal lengths and exposure amounts. depth of field A technology has been proposed for generating EDOF+HDR images with expanded optical distance and dynamic range. In Patent Document 1, a near-point image with low exposure and a far-point image with high exposure are acquired by adjusting the ratio of the exposure amounts of the near-point image and the far-point image using a dimming mirror. In Patent Document 2, multiple images with different optical distances and brightness are acquired by splitting the incident light. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5856733 [Patent Document 2] International Publication No. 2018 / 221041 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Documents 1 and 2 disclose a hardware configuration for generating an EDOF+HDR image, but do not disclose a specific method for synthesizing multiple images to generate an EDOF+HDR image. The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image processing device, an image processing method, and a recording medium that can generate an EDOF+HDR image in which multiple images are combined at an appropriate combination ratio. [Means for solving the problem]

[0006] One aspect of the present invention is a method for detecting an object by a processor, the method comprising: receiving a far point image focused on a far point and a near point image focused on a near point; and the far point image and the near point image having different exposure amounts. generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; The synthesis ratio of each region of the far point image and the near point image is The determination map and Brightness or frequency The aforementioned and a predetermined relationship between the composite ratio and the calculated Before The image processing device synthesizes the far point image and the near point image using the synthesis ratio.

[0007] Another aspect of the present invention is a method for receiving a far point image focused on a far point and a near point image focused on a near point, the far point image and the near point image having different exposure amounts; generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; The synthesis ratio of each region of the far point image and the near point image is The determination map and Brightness or frequency The aforementioned and a predetermined relationship between the composite ratio and the calculated Before The image processing method uses the synthesis ratio to synthesize the far point image and the near point image.

[0008] Another aspect of the present invention is a computer-readable non-transitory recording medium having an image processing program recorded thereon, the image processing program including: receiving a far point image focused on a far point and a near point image focused on a near point; the far point image and the near point image have different exposure amounts; generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; The synthesis ratio of each region of the far point image and the near point image is The determination map andBrightness or frequency The aforementioned and a predetermined relationship between the composite ratio and the calculated Before The recording medium causes a computer to synthesize the far point image and the near point image using the synthesis ratio. [Effects of the Invention]

[0009] According to the present invention, it is possible to generate an EDOF+HDR image in which a plurality of images are combined at an appropriate combination ratio. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention; [Figure 2A] 10 is a graph showing an example of the relationship between brightness and a blending ratio. [Figure 2B] 10 is a graph showing an example of the relationship between frequency and synthesis ratio. [Figure 2C] 10 is a graph showing another example of the relationship between frequency and synthesis ratio. [Figure 3A] 1 is a flowchart of an image processing method according to an embodiment of the present invention. [Figure 3B] 3B is a flowchart showing step S2 in the image processing method of FIG. 3A. [Figure 4A] FIG. 10 is a diagram showing an example of a far point image or a near point image. [Figure 4B] FIG. 4B is a diagram showing a decision map generated from the image of FIG. 4A. [Figure 5] 10A and 10B are diagrams illustrating another method for calculating the blending ratio of each region. [Figure 6A] 10 is a flowchart of an image processing method according to another embodiment of the present invention. [Figure 6B] 6B is a flowchart showing steps S4 and S5 in the image processing method of FIG. 6A. [Figure 7] 10 is a graph showing another example of the relationship between frequency and synthesis ratio. [Figure 8]10 is a graph showing another example of the relationship between brightness and blending ratio. [Figure 9] 10A and 10B are diagrams illustrating a method for adjusting the relationship between brightness and a blending ratio. DETAILED DESCRIPTION OF THE INVENTION

[0011] An image processing apparatus, an image processing method, and a recording medium according to an embodiment of the present invention will be described below with reference to the drawings. The image processing device 1 according to this embodiment synthesizes a plurality of images, and compares the images with each other. depth of field 1, the image processing device 1 includes a processor 2 such as a central processing unit, a memory 3, a storage unit 4, and an input / output unit 5.

[0012] The image processing device 1 is incorporated as part of, for example, an endoscope system. The input / output unit 5 has a known input / output interface, and the image processing device 1 receives images acquired by the endoscope 10 through the input / output unit 5. The endoscope 10 has an imaging element 10a such as a CMOS image sensor or a CCD image sensor, and acquires a far point image and a near point image of the subject by the imaging element 10a. The far point image is an image focused on a far point, and the near point image is an image focused on a near point. Therefore, the far point image and the near point image have different focal lengths. Furthermore, the far point image and the near point image have different amounts of exposure.

[0013] The far point image and the near point image, which differ from each other in both focal length and exposure amount, are acquired using known means. In one example, the far point image and the near point image are sequentially captured by the image sensor 10a by continuously capturing images of the subject under different exposure conditions, such as the amount of light from the light source, the shutter speed of the image sensor 10a, or the gain.

[0014] In another example, the far-point image and the near-point image are simultaneously acquired by the image sensor 10a capturing a single image of the subject. In this case, the imaging optical system of the endoscope 10 may include a prism in front of the image sensor 10a that splits light from the subject into two beams. The prism gives the two beams different optical path lengths, and one or two image sensors 10a simultaneously capture the two beams. To achieve different exposure levels, the prism may split light from the subject into two beams of light with different intensities. The light split ratio is adjusted, for example, by a coating applied to the prism. Alternatively, the imaging optical system may include two circuits connected to each pixel of the imaging element 10a. Each pixel outputs a signal to the two circuits, and an image without gain is acquired from one circuit and an image with gain is acquired from the other circuit.

[0015] The memory 3 is made up of a volatile storage device such as a RAM, and is used as a working area for the processor 2 . The storage unit 4 is composed of a computer-readable non-transitory recording medium such as a ROM, a flash memory, or a hard disk drive.

[0016] The storage unit 4 stores an image processing program 4a for causing the processor 2 to execute an image processing method described below. Furthermore, as shown in FIGS. 2A, 2B, and 2C, the storage unit 4 stores predetermined relationships 6 and 7 used to combine a far-point image and a near-point image in the image processing method. FIG. 2A shows an example of the predetermined relationship 6 between brightness and combining ratio, and FIGS. 2B and 2C each show an example of the predetermined relationship 7 between frequency and combining ratio. The relationships 6 and 7 are experimentally determined in advance based on, for example, an image of a subject acquired by an endoscope. In this specification, frequency refers to the spatial frequency of the image.

[0017] Next, the image processing method executed by the processor 2 will be described. As shown in FIG. 3A, the image processing method includes step S1 of receiving a far point image and a near point image, step S2 of calculating a synthesis ratio between the far point image and the near point image, and step S3 of synthesizing the far point image and the near point image based on the synthesis ratio to generate an EDOF+HDR image.

[0018] 3B, step S2 includes steps S21, S22, and S23. When a pair of a far point image and a near point image is received from the endoscope 10 (step S1), the processor 2 selects one of the far point image and the near point image (step S21). In step S21, the processor 2 may select an image having fewer blown-out or crushed-out pixels whose brightness values ​​are saturated, or may select a preset image.

[0019] Next, processor 2 generates a decision map representing the brightness of each region in the selected image (step S22). Specifically, processor 2 generates a luminance image of selected image A, calculates the low-frequency components of the luminance image by applying a known low-pass filter such as a Gaussian filter to the luminance image, and generates a decision map representing the low-frequency components of each region in image A. Fig. 4A shows a far-point image or a near-point image that is selected image A, and Fig. 4B shows decision map B generated from image A of Fig. 4A. Decision map B is an image that represents the rough spatial changes in brightness in image A, with fine spatial changes in brightness based on the fine structure of the subject removed.

[0020] Next, processor 2 calculates the blending ratio of each region R of the far point image and the near point image based on decision map B and relationship 6 (step S23). Region R is a region consisting of one pixel or multiple pixels. If region R consists of multiple pixels, the value of region R is, for example, the average of the values ​​of the multiple pixels. Specifically, processor 2 calculates the blending ratio corresponding to the value (brightness) of the low-frequency component of each region R in decision map B from relationship 6. In a scene of a lumen such as the intestinal tract, regions with a short object distance are bright, and regions with a long object distance are dark. The object distance is the distance in the optical axis direction from the tip of endoscope 10 to the object when the image is acquired. In relationship 6, the blending ratio of the near point image increases as the brightness increases, and the blending ratio of the far point image increases as the brightness decreases (see FIG. 2A).

[0021] Next, the processor 2 combines the far point image and the near point image using the combination ratio of each region R calculated in step S23 (step S3). Specifically, the processor 2 combines the pixel values ​​of each pixel in the region R at the same position of the far point image and the near point image using the calculated combination ratio. depth of field EDOF processing, which expands the image quality, and HDR processing, which expands the dynamic range, are performed simultaneously to generate an EDOF+HDR image. After step S3, the processor 2 may output the EDOF+HDR image to an external device connected to the image processing device 1, such as a display device.

[0022] As described above, according to this embodiment, the blending ratio of each region R is calculated based on the brightness of one of the far-point image and the near-point image. The brightness of an image of an object such as a lumen correlates with the object distance, and in the far-point image and the near-point image, areas at long object distances are darker and areas at short object distances are brighter. Therefore, an appropriate blending ratio for each region R can be calculated based on the brightness. Furthermore, using such an appropriate blending ratio, a high-quality EDOF+HDR image that is well-focused over a wide depth of field can be generated.

[0023] Furthermore, according to this embodiment, the blending ratio is calculated based on the low-frequency components of the image brightness. The low-frequency components, from which minute variations in brightness due to the detailed structure of the subject have been removed, represent a more accurate subject distance. Therefore, an appropriate blending ratio can be stably calculated based on the low-frequency components. Furthermore, according to this embodiment, the low-frequency components of all regions R in image A are calculated at once by generating determination map B. This reduces the amount of calculation and time required to calculate the blending ratios of all regions R.

[0024] In this embodiment, the processor 2 may generate a judgment map representing the frequency of the image A, instead of the judgment map representing the brightness of the image A. In this case, processor 2 generates a luminance image of image A, calculates the high-frequency components of the luminance image by applying a known high-pass filter to the luminance image, and generates a judgment map representing the high-frequency components of each region R within image A (step S22).

[0025] Next, the processor 2 calculates the synthesis ratio of each region R based on the value of the high frequency component of each region R in the determination map and the relationship 7 (step S23). When using a decision map that represents frequency components, the conditions differ between a far point image and a near point image. If the selected image A is a far-point image, in the lumen scene, the object structure is sparse in the area of ​​the short object distance and dense in the area of ​​the long object distance. Therefore, in relation 7, the higher the frequency, the higher the synthesis ratio of the far-point image, and the lower the frequency, the higher the synthesis ratio of the near-point image (see Figure 2B). In this way, the frequency of the image correlates with the subject distance, and in the far-point image, the high-frequency components are more abundant in areas with long subject distances and less abundant in areas with short subject distances. Therefore, it is possible to calculate an appropriate blending ratio for each region R based on the frequency.

[0026] On the other hand, if the selected image A is a near-point image, in the lumen scene, the object structure is dense in the area of ​​the short object distance and sparse in the area of ​​the long object distance. Therefore, in relation 7, the higher the frequency, the higher the synthesis ratio of the near-point image, and the lower the frequency, the higher the synthesis ratio of the far-point image (see Figure 2C). In this way, the frequency of the image correlates with the object distance, and in the near point image, the high frequency components are fewer in areas with long object distances and more in areas with short object distances. Therefore, it is possible to calculate an appropriate blending ratio for each area R based on the frequency.

[0027] In this embodiment, the processor 2 generates a determination map. Alternatively, the processor 2 may execute a process of calculating a blending ratio for each region R without generating a determination map. For example, as shown in FIG. 5, processor 2 sets one region R in image A, calculates the low-frequency or high-frequency component of region R, and calculates the blending ratio of region R based on the low-frequency or high-frequency component. Next, processor 2 moves region R within image A and calculates the blending ratio of region R after the movement. FIG. 5 shows an example in which region R is scanned using a raster method. In this way, processor 2 can calculate an appropriate blending ratio for each region R by calculating the blending ratio for each region R while scanning region R within image A.

[0028] In the above embodiment, as shown in FIG. 6A, the processor 2 may determine the subject distance of the far point image or the near point image (step S4), and set the relationships 6 and 7 used to calculate the synthesis ratio according to the subject distance (step S5).

[0029] Specifically, as shown in FIG. 6B, in step S4, processor 2 determines the distribution of subject distance based on the distribution of brightness of image A (step S41). For example, processor 2 calculates a brightness histogram of image A. If the distribution of subject distance is wide (for example, image A contains both close and far-distance subjects), the brightness of image A will be distributed over a wide range, and the width of the histogram will be wide. On the other hand, if the distribution of subject distance of image A is narrow (for example, image A contains only one of close and far-distance subjects), the brightness of image A will be distributed unevenly, and the width of the histogram will be narrow.

[0030] If the distribution of subject distances is wide (for example, the width of the histogram is equal to or greater than a predetermined threshold) (YES in step S42), processor 2 selects relationship 6 for the far point and near point (step S51). As described above, relationship 6 is the relationship between brightness and the blending ratio. In this case, processor 2 calculates the blending ratio based on brightness and relationship 6 (step S2).

[0031] On the other hand, if the distribution of the object distance is narrow (for example, the width of the histogram is less than a predetermined threshold) (NO in step S42), the processor 2 then determines whether the object distance is long or not based on the frequencies of both the far point image and the near point image (step S43). Specifically, the processor 2 calculates the high frequency components of each of the far point image and the near point image, and compares the high frequency components of the far point image with the high frequency components of the near point image.

[0032] If the far-point image contains more high-frequency components than the near-point image, processor 2 determines that the subject distance is long (YES in step S43) and selects relationship 7a for the far point (step S52). As shown in FIG. 7, relationship 7a is a relationship between frequency and blending ratio, and in relationship 7a, the blending ratio of the far-point image is higher than that of the near-point image across all frequencies; for example, the blending ratio of the far-point image is 100% across all frequencies. In this case, processor 2 calculates the blending ratio based on the frequency and relationship 7a (step S2). If the blending ratio of the far-point image is 100%, an EDOF+HDR image substantially consisting of the far-point image is generated (step S3).

[0033] On the other hand, if the near point image contains more high-frequency components than the far point image, processor 2 determines that the subject distance is short (NO in step S43) and selects relationship 7b for the near point (step S53). As shown in FIG. 7, relationship 7b is a relationship between frequency and blending ratio, and in relationship 7b, the blending ratio of the near point image is higher than that of the far point image across all frequencies; for example, the blending ratio of the near point image is 100% across all frequencies. In this case, processor 2 calculates the blending ratio based on the frequency and relationship 7b (step S2). If the blending ratio of the near point image is 100%, an EDOF+HDR image substantially consisting of the near point image is generated (step S3).

[0034] During observation of the inside of a body cavity with the endoscope 10, the image scene changes, and the distribution of object distances also changes in response to the change in the scene. For example, in a scene in which a lumen is observed in the longitudinal direction as shown in Figure 4A, object distances are distributed over a wide range from close to long distances. In a scene in which the tip of the endoscope 10 is brought close to the inner wall of the body cavity for observation, the object distances are distributed biased toward only close distances. 6A and 6B, the relationship 6, 7a, or 7b is set based on the distribution of subject distances, depending on the scene in the image. This allows the calculation of an appropriate blending ratio depending on the scene, enabling the generation of a high-quality EDOF+HDR image. After step S43, the processor 2 may output the far point image or the near point image as an EDOF+HDR image without performing steps S5, S2, and S3.

[0035] Furthermore, when the distribution of object distances is narrow, the object distance is further determined based on frequency. By determining the object distance based on both brightness and frequency in this way, an appropriate blending ratio can be calculated robustly for various scenes. The distribution of the object distance is determined using the brightness and frequency of the far-point image and the near-point image. That is, the object distance can be determined by calculation alone, without the need for a device such as a sensor for measuring the object distance.

[0036] If the distribution of subject distances is wide (YES in step S42), in step S51, as shown in FIGS. 8 and 9, processor 2 may set relationship 6 in more detail according to the distribution of subject distances. 8, processor 2 selects one of multiple prepared relationships 6a, 6b, and 6c, depending on, for example, the width of the histogram. For example, if the histogram is biased toward bright areas, relationship 6a is selected. 9, processor 2 changes relationship 6 according to the width of the histogram, etc., by moving intersection P between the graph of the composition ratio of the far-point image and the graph of the composition ratio of the near-point image in the direction of brightness. When intersection P moves to the end of the graph, relationship 6 becomes relationship 7a or 7b shown in FIG.

[0037] Although the embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and various improvements are possible without departing from the spirit of the present invention. For example, the image processing device 1 may process far point images and near point images acquired by an imaging device other than the endoscope 10, such as a digital camera or a microscope, or may generate an EDOF+HDR image by combining three or more images with different focal lengths and exposure amounts. [Explanation of symbols]

[0038] 1. Image processing device 2 processors 4. Memory unit (recording medium) 6,6a,6b,6c Relationship between brightness and blending ratio 7,7a,7b Relationship between frequency and synthesis ratio A Far point image, near point image B. Judgment Map R area

Claims

1. a processor; The processor: receiving a far point image focused on a far point and a near point image focused on a near point, the far point image and the near point image having different exposure amounts; generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; calculating a blending ratio of each region of the far point image and the near point image based on the determination map and a predetermined relationship between brightness or frequency and the blending ratio; an image processing device that combines the far point image and the near point image using the calculated combination ratio;

2. the processor: determining a subject distance of the far point image or the near point image; The image processing device according to claim 1 , wherein the relationship used to calculate the blending ratio is set based on the subject distance.

3. The image processing device according to claim 2 , wherein the processor sets the relationship by selecting one from a plurality of the relationships depending on the subject distance.

4. The image processing device according to claim 2 , wherein the processor sets the relationship by changing the relationship between the brightness or frequency and the blending ratio in the relationship according to the subject distance.

5. the processor: calculating a low frequency component of one of the far point image and the near point image; generating a decision map representing the calculated low-frequency components as the decision map; The image processing device according to claim 1 , wherein the combining ratio is calculated based on a determination map representing the calculated low-frequency components and the relationship between brightness and the combining ratio.

6. the processor: calculating a high frequency component of one of the far point image and the near point image; generating a decision map representing the calculated high frequency components as the decision map; The image processing device according to claim 1 , wherein the combining ratio is calculated based on a determination map representing the calculated high frequency components and the relationship between a frequency and the combining ratio.

7. receiving a far point image focused on a far point and a near point image focused on a near point, the far point image and the near point image having different exposure amounts; generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; calculating a blending ratio of each region of the far point image and the near point image based on the determination map and a predetermined relationship between brightness or frequency and the blending ratio; an image processing method for combining the far point image and the near point image using the calculated combination ratio;

8. A computer-readable non-transitory recording medium on which an image processing program is recorded, The image processing program receiving a far point image focused on a far point and a near point image focused on a near point, the far point image and the near point image having different exposure amounts; generating a decision map representing the brightness or frequency of each region of one of the far point image and the near point image; calculating a blending ratio of each region of the far point image and the near point image based on the determination map and a predetermined relationship between brightness or frequency and the blending ratio; A recording medium that causes a computer to synthesize the far point image and the near point image using the calculated synthesis ratio.

Images