Image processing apparatus and image processing method

Abstract

To suppress generation of a false color in an interpolated image by correctly detecting an edge included in an image, and interpolating pixel values based on the detected edge.SOLUTION: An image processing device which replaces pixel values received from an imaging device that includes multiple types of pixels different in wavelength region of light to be detected, with pixel values of pixels in a second pixel arrangement different from a first pixel arrangement of the imaging device includes: a first determination unit which determines whether there is an edge indicating a direction along which pixels having a smaller change in pixel values are arranged, out of peripheral pixels located around a pixel of interest whose pixel value is to be interpolated and which has the same color as the pixel of interest after replacement; and an interpolation processing unit which interpolates, if an edge is determined, a pixel value of the pixel of interest using the pixel values of the peripheral pixels along the direction of the edge, or if no edge, interpolates the pixel value of the pixel of interest using pixel values of pixels along a direction of an edge detected based on luminance of the pixels around the pixel of interest.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] In vehicle-mounted cameras and the like, lighting conditions can vary greatly depending on the location or time of day when a subject is photographed. To obtain appropriate images under various lighting conditions, imaging devices have been developed that include, in addition to red, green, and blue color filters, an infrared filter that transmits infrared light or a white filter that transmits all of the red, green, and blue colors. The pixel arrangement of this type of imaging device differs from existing pixel arrangements such as the Bayer array. Therefore, in order to perform image processing using existing processing techniques, it is necessary to convert the image data acquired by the imaging device into Bayer array image data by interpolating pixel values. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-259060 [Patent Document 2] Japanese Patent Application Publication No. 2019-106576 [Patent Document 3] Japanese Patent Application Laid-Open No. 2008-258932 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, the interpolation process of pixel values is performed by detecting an edge where the luminance changes significantly around the pixel of interest and using the pixel values of pixels of the same color as the pixel of interest arranged along the direction of the edge. However, for example, when the difference in luminance between two image regions with different color tones adjacent to each other is small, it may not be possible to detect the edge at the boundary between the two image regions. If the interpolation process of pixel values considering the edge is not performed even though the edge exists, there is a possibility that false colors (artifacts) different from the color of the original image will occur in the interpolated image.

[0005] The present invention has been made in view of the above points, and an object thereof is to correctly detect an edge included in an image and suppress the occurrence of false colors in the interpolated image by interpolating pixel values based on the detected edge.

Means for Solving the Problems

[0006] In one aspect of the present invention, an image processing apparatus replaces pixel values received from an imaging device including a plurality of types of pixels having different wavelength regions of light to be detected with pixel values of pixels in a second pixel array different from the first pixel array of the imaging device. A first determination unit that determines the presence or absence of an edge indicating a direction in which pixels with little change in pixel value are arranged among the peripheral pixels having the same color as the color of the pixel of interest after replacement of the pixel of interest, which are located around the pixel of interest for which pixel values are to be interpolated; when it is determined that there is an edge, the pixel value of the pixel of interest is interpolated using the pixel values of the pixels along the direction of the edge among the peripheral pixels, and when it is determined that there is no edge, the pixel value of the pixel of interest is interpolated using the pixel values of the pixels along the direction of the edge detected based on the luminance of the pixels located around the pixel of interest If no edge is detected even based on the luminance of the pixels located around the target pixel, the pixel value of the target pixel is interpolated using all of the surrounding pixels located around the target pixel and having the same color as the color after replacement of the target pixel and an interpolation processing unit for performing the interpolation.

Effects of the Invention

[0007] According to the disclosed technology, it is possible to correctly detect an edge included in an image and suppress the occurrence of false colors in the interpolated image by interpolating pixel values based on the detected edge.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description, image data may sometimes be simply referred to as an image.

[0010] (First Embodiment) FIG. 1 shows an example image of an image processing system including an image processing apparatus according to the first embodiment. The image processing system 100 shown in FIG. 1 is mounted on a moving body 200 such as an automobile, for example. Imaging devices 19A, 19B, 19C, 19D, and 19E such as cameras are installed in front of, behind, to the left, and to the right of the moving body 200 in the traveling direction D, and in front of the interior of the moving body 200. Hereinafter, when the imaging devices 19A, 19B, 19C, 19D, and 19E are described without distinction, they are also referred to as the imaging device 19. An example of the pixels of the image sensor mounted on the imaging device 19 is described with reference to FIG. 6.

[0011] Note that the number and installation positions of the imaging devices 19 installed in the moving body 200 are not limited to those shown in FIG. 1. For example, one imaging device 19 may be installed only in front of the moving body 200, or two imaging devices 19 may be installed only in the front and the rear. Alternatively, the imaging device 19 may be installed on the ceiling of the moving body 200.

[0012] In addition, the moving body 200 on which the image processing system 100 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory. Further, the image processing system 100 may be a system that processes images acquired from imaging devices other than the imaging device 19 installed in the moving body 200, such as a surveillance camera, a digital still camera, or a digital camcorder.

[0013] Each imaging device 19 is connected to the image processing apparatus 10 via a signal line or wirelessly. Further, the distance between each imaging device 19 and the image processing apparatus 10 may be longer than the distance imaged in FIG. 1. For example, the image data acquired by the imaging device 19 may be transmitted to the image processing apparatus 10 via a network. In this case, at least one of the image processing apparatus 10 and the information processing apparatus 11 may be realized by cloud computing.

[0014] The image processing system 100 includes an image processing device 10, an information processing device 11, and a display device 12. In FIG. 1, for the sake of easy understanding of the description, the image processing system 100 is superimposed and shown on an image diagram of the moving body 200 viewed from above. However, in reality, the image processing device 10 and the information processing device 11 are mounted on a control board or the like mounted on the moving body 200, and the display device 12 is installed at a position visible to a person such as a driver inside the moving body 200. Note that the image processing device 10 may be mounted on a control board or the like as a part of the information processing device 11.

[0015] FIG. 2 shows an example of the functional configuration of the image processing device 10 in FIG. 1. The image processing device 10 includes an acquisition unit 10a, a first determination unit 10b, a second determination unit 10c, an interpolation processing unit 10d, and an output unit 10e. The acquisition unit 10a acquires image data indicating an image around the moving body 200 captured by each imaging device 19.

[0016] The first determination unit 10b uses the image data acquired from the acquisition unit 10a to determine the presence or absence of an edge indicating the direction in which pixels with little change in pixel value are arranged among the peripheral pixels of the target pixel that have the same color as the color after replacement of the target pixel and are located around the target pixel for which the pixel value is to be interpolated (first determination process). The second determination unit 10c uses the image data acquired from the acquisition unit 10a to determine an edge based on the luminance of the pixels located around the target pixel (second determination process). For example, the determination of the edge by the second determination unit 10c may be performed before the determination of the presence or absence of the edge by the first determination unit 10b, or may be performed when the first determination unit 10b determines that there is no edge.

[0017] When the first determination unit 10b determines that there is an edge, the interpolation processing unit 10d interpolates the pixel value of the target pixel using the pixel values of the pixels along the direction of the edge among the peripheral pixels. When the first determination unit 10b determines that there is no edge, the interpolation processing unit 10d interpolates the pixel value of the target pixel using the pixel values of the pixels along the direction of the edge based on the luminance of the pixels determined by the second determination unit 10c.

[0018] Accordingly, the pixel value interpolation process based on the edge detected based on the difference in pixel values (color difference) can be preferentially performed over the pixel value interpolation process based on the edge detected based on the luminance difference. Therefore, even when the luminance difference between two adjacent image regions with different colors is small and the edge at the boundary between the two image regions cannot be detected by the luminance difference, the pixel value interpolation process can be appropriately performed based on the edge detected by the color difference. As a result, it is possible to suppress the pixel value interpolation process from being performed across the boundary of the image region, and it is possible to suppress the occurrence of false colors at the boundary of the image region.

[0019] The output unit 10e outputs the interpolated image data including the pixel values of the pixels interpolated by the interpolation processing unit 10d to at least one of the display device 12 and the information processing device 11 as an image processing result.

[0020] FIG. 3 shows an outline of the configuration of various devices mounted on the moving body 200 of FIG. 1. The moving body 200 includes an image processing device 10, an information processing device 11, a display device 12, at least one ECU (Electronic Control Unit) 13, and a wireless communication device 14 that are interconnected via an internal network. The moving body 200 also includes a sensor 15, a drive device 16, a lamp device 17, a navigation device 18, and an imaging device 19. For example, the internal network is an in-vehicle network such as a CAN (Controller Area Network) or Ethernet (registered trademark).

[0021] The image processing device 10 receives the image data (frame data) acquired by the imaging device 19 and performs image processing using the received image data. The information processing device 11 performs processing such as image recognition using the image data image-processed by the image processing device 10. For example, the information processing device 11 may recognize an object such as a person, a signal, or a sign outside the moving body 200 based on the image generated by the image processing device 10, and may track the recognized object. The information processing device 11 may function as a computer that controls each part of the moving body 200. Further, the information processing device 11 may control the entire moving body 200 by controlling the ECU 13.

[0022] The display device 12 displays an image, a corrected image, etc. using the image data generated by the image processing device 10. The display device 12 may display the image in the reverse direction of the moving body 200 in real time when the moving body 200 moves backward (backs up). Further, the display device 12 may display the image output from the navigation device 18.

[0023] The ECU 13 is provided corresponding to each mechanism part such as an engine or a transmission. Each ECU 13 controls the corresponding mechanism part based on an instruction from the information processing device 11. The wireless communication device 14 communicates with a device outside the moving body 200. The sensor 15 is a sensor that detects various types of information. The sensor 15 may include, for example, a position sensor that acquires the current position information of the moving body 200. Further, the sensor 15 may include a speed sensor that detects the speed of the moving body 200.

[0024] The drive device 16 is various devices for moving the moving body 200. The drive device 16 may include, for example, an engine, a steering device (steering), and a braking device (brake), etc. The lamp device 17 is various lamps mounted on the moving body 200. The lamp device 17 may include, for example, headlamps (headlamps, headlights), lamps of direction indicators (indicators), backlights, and brake lamps, etc. The navigation device 18 is a device that guides the route to the destination by voice and display.

[0025] The imaging device 19 has, for example, an RGBIr image sensor IMGS equipped with pixels including a plurality of types of filters that transmit red light R, green light G, blue light B, and near-infrared light Ir respectively. That is, the image sensor IMGS includes a plurality of types of pixels with different wavelength regions of the light to be detected.

[0026] The pixels that detect the near-infrared light Ir are an example of pixels other than red, green, and blue. The image sensor IMGS may have not only one type of other pixel other than RGB, but also a plurality of types of other pixels. Note that the image sensor IMGS may have pixels that detect light in other wavelength regions (for example, all of RGB) instead of the pixels that detect the near-infrared light Ir.

[0027] As described above, the image data acquired by the imaging device 19 is processed by the image processing device 10. For example, the image processing device 10 corrects (interpolates) the image data acquired by the RGBIr image sensor IMGS and generates image data in a Bayer array. The image processing performed by the image processing device 10 will be described with reference to FIGS. 5 to 7. The RGBIr pixel array of the image sensor IMGS is an example of a first pixel array. The pixel array in the Bayer array is an example of a second pixel array.

[0028] Note that the image processing device 10 may convert the image data acquired by a Bayer array image sensor into image data other than the Bayer array. Furthermore, the image processing device 10 may record the image data generated by correction in an external or internal recording device.

[0029] FIG. 4 shows an example of the configuration of the image processing apparatus 10 and the information processing apparatus 11 in FIG. 3. Since the configurations of the image processing apparatus 10 and the information processing apparatus 11 are the same as each other, the configuration of the image processing apparatus 10 will be described below. For example, the image processing apparatus 10 includes a CPU 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a memory device 24 that are interconnected by a bus BUS.

[0030] The CPU 20 executes an image processing program stored in the memory device 24 to perform various image processes described later. The interface device 21 is used to connect to a network (not shown). The auxiliary storage device 23 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc., and holds an image processing program, image data, and various parameters used for image processing.

[0031] The memory device 24 is, for example, a DRAM (Dynamic Random Access Memory), etc., and holds an image processing program transferred from the auxiliary storage device 23. The drive device 22 has an interface for connecting a recording medium 30, and transfers, for example, an image processing program stored in the recording medium 30 to the auxiliary storage device 23 based on an instruction from the CPU 20. Note that the drive device 22 may transfer image data, etc. stored in the auxiliary storage device 23 to the recording medium 30.

[0032] FIG. 5 shows an example of converting the RGBIr image data IMG1 acquired by the image sensor IMGS in FIG. 3 into Bayer array image data IMG2. The image sensor IMGS has a pixel R for detecting red light, a pixel G for detecting green light, a pixel B for detecting blue light, and a pixel Ir for detecting near-infrared light.

[0033] In a unit of repetition of the array of the image sensor IMGS, in a vertical 4 pixels and horizontal 4 pixels, pixels R, G, B, and Ir are arranged in a ratio of 1:4:1:2. Pixel Ir is arranged at the position of pixel R or pixel B in the Bayer array. The pixel array shown in the lower left bracket of FIG. 5 is obtained by swapping pixel R and pixel B in the pixel array of the upper left (the pixel positions are shifted). Hereinafter, the pixel array of the image sensor IMGS is also referred to as a non-Bayer array.

[0034] On the other hand, in the Bayer array shown in the upper right of FIG. 5, in a vertical 2 pixels and horizontal 2 pixels, pixels R, G, and B are arranged in a ratio of 1:2:1. In the non-Bayer arrays in the upper left and lower left of FIG. 5, the pixels indicated by the thick frame indicate that the colors are different from the pixels in the Bayer array. When converting the image data IMG1 of the non-Bayer array into the image data IMG2 of the Bayer array, the pixel values of the pixels in the Bayer array corresponding to the pixels of the thick frame are generated by performing interpolation processing using the pixel values of the surrounding pixels of the same color.

[0035] In one method of pixel value interpolation processing, first, based on the luminance of each pixel obtained from the RGB pixel values, the position (edge direction) of an edge, which is a boundary portion of an image where the luminance changes greatly, is detected. The pixel of interest for which the pixel value is to be interpolated belongs to one of the image regions bisected by the edge. Then, the pixel value is interpolated using the pixel values of the pixels included in the same image region as the pixel of interest and along the direction of the edge. Alternatively, the pixel value is interpolated using the pixel values of the same-color pixels included in the same image region as the pixel of interest and close to the position of the pixel of interest. By not using the pixel values of the pixels straddling the edge for interpolation, it becomes possible to appropriately interpolate the pixel values.

[0036] However, as shown in the lower right of FIG. 5, for example, when a blue image region 1 and a red image region 2 are adjacent, in the above-described interpolation of pixel values based on luminance, false colors (artifacts) may occur at the boundary of the image regions. Due to the occurrence of false colors, an image in which the boundary between the image regions 1 and 2 is blurred is generated. The cause of the occurrence of false colors will be explained below.

[0037] In the image area 1, let the pixel value of pixel R, the pixel value of pixel G, and the pixel value of pixel B be R1, G1, and B1 respectively. In the image area 2, let the pixel value of pixel R, the pixel value of pixel G, and the pixel value of pixel B be R2, G3, and B3 respectively. Also, for the sake of easy understanding of the explanation, in each of the image area 1 and the image area 2, it is assumed that the components of each of the RGB colors are uniform.

[0038] In the blue image area 1, the pixel value B1 is large and the pixel value R1 is small. In the red image area 2, the pixel value R2 is large and the pixel value B2 is small. Assume that the pixel value B1 is approximately equal to the pixel value R2, and the pixel value R1 is approximately equal to the pixel value B2. Assume that the pixel value G1 in the image area 1 and the pixel value G2 in the image area 2 are approximately equal.

[0039] The luminance Y1 of the image area 1 and the luminance Y2 of the image area 2 can be calculated by, for example, equations (1) and (2) respectively. The symbol "*" in the equations indicates the multiplication symbol. Luminance Y1 = (pixel value R1 + pixel value G1 * 2 + pixel value B1) / 4 …(1) Luminance Y2 = (pixel value R2 + pixel value G2 * 2 + pixel value B2) / 4 …(2)

[0040] As described above, when the pixel value B1 is approximately equal to the pixel value R2, the pixel value R1 is approximately equal to the pixel value B2, and the pixel value G1 is approximately equal to the pixel value G2, the luminances Y1 and Y2 calculated from equations (1) and (2) are approximately equal. When the luminances Y1 and Y2 are approximately equal, in the method of determining the edge direction based on the luminance difference, the edge that is the boundary between the image area 1 and the image area 2 cannot be detected.

[0041] Therefore, when calculating the pixel value of a pixel near the boundary with the image area 2 in the image area 1 by interpolation, not only the pixel value of the pixel in the image area 1 but also the pixel value of the pixel in the image area 2 is referred to. Similarly, when calculating the pixel value of a pixel near the boundary with the image area 1 in the image area 2 by interpolation, not only the pixel value of the pixel in the image area 2 but also the pixel value of the pixel in the image area 1 is referred to. As a result, false colors occur in the boundary portion between the image area 1 and the image area ②.

[0042] FIG. 6 shows an example of a method for detecting an edge using the image data acquired by the image sensor IMGS of FIG. 3, and an example of a method for interpolating pixel values based on the edge detection result. The 25 pixels shown in FIG. 6 show a part of the pixels mounted on the image sensor IMGS.

[0043] In FIG. 6, for the sake of easy explanation, consecutive numbers are assigned to each color for pixels R, G, B, and Ir. The symbol "->B" enclosed in parentheses for pixel R indicates that the pixel value of pixel R is replaced by the pixel value of pixel B through an interpolation process for converting the pixel value of the Bayer array image data. The symbol "->R" enclosed in parentheses for pixel Ir indicates that the pixel value of pixel Ir is replaced by the pixel value of pixel R through an interpolation process for converting the pixel value of the Bayer array image data.

[0044] Hereinafter, an example is shown in which the pixel value of the central pixel R3 indicated by the thick frame among the 25 pixels is replaced by the pixel value of pixel B in the Bayer array. The edge detection process and the pixel value interpolation process are realized by the CPU 20 (FIG. 4) mounted on the image processing apparatus 10 of FIG. 3 executing an image processing program to implement an image processing method. Hereinafter, the pixel R3 indicated by the thick frame and the pixel B after replacement are also referred to as the pixel of interest.

[0045] First, the image processing apparatus 10 generates color difference data (gradient) of pixel B in each direction using the pixel values of four pixels B1 - B4 of the same type as the pixel of interest B, which are located on both sides in the horizontal direction H and both sides in the vertical direction V with respect to the pixel of interest R3. Hereinafter, the pixel value of each of the pixels B1 - B4 is referred to as pixel values B1 - B4. The horizontal direction H is an example of the first direction, and the vertical direction V is an example of the intersecting direction of the first direction.

[0046] The color difference data gradH in the horizontal direction H, the color difference data gradV in the vertical direction V, the color difference data gradN in the diagonal direction from the upper left to the lower right, and the color difference data gradZ in the diagonal direction from the lower left to the upper right are calculated by formulas (3), (4), (5), and (6). In formulas (3) to (6), the symbol abs indicates the calculation of the absolute value. Also, the calculation results of formulas (5) and (6) may have the decimal part truncated.

[0047] gradH = abs(B2 - B3) …(3) gradV = abs(B1 - B4) …(4) gradN = (abs(B1 - B3) + abs(B4 - B2)) / 2 …(5) gradZ = (abs(B1 - B2) + abs(B4 - B3)) / 2 …(6)

[0048] Next, the image processing apparatus 10 calculates the minimum gradient value minGrad among the color difference data gradH, gradV, gradN, and gradZ according to formula (7). minGrad = min(gradH, gradV, gradZ, gradN) …(7) In formula (7), the symbol min(gradH, gradV, gradZ, gradN) indicates a function for calculating the minimum value.

[0049] Then, when the minimum gradient value minGrad is the color difference data gradH and the difference (absolute value) between the color difference data gradH and gradZ is greater than or equal to a predetermined threshold T, the image processing apparatus 10 determines that the horizontal direction H is the edge direction. When the minimum gradient value minGrad is the color difference data gradV and the difference (absolute value) between the color difference data gradH and gradZ is greater than or equal to a predetermined threshold T, the image processing apparatus 10 determines that the vertical direction V is the edge direction. Then, the image processing apparatus 10 performs an interpolation process of interpolating the pixel value of the target pixel B using the pixel values of other pixels B based on the edge direction determined by the color difference.

[0050] When the difference (absolute value) between the color difference data gradH and gradZ is smaller than a predetermined threshold value T, there is a possibility that the accuracy of determining the direction of the edge due to the color difference is low. In this case, the image processing apparatus 10 does not determine the direction of the edge based on the color difference, but determines the direction of the edge based on the luminance difference. Then, the image processing apparatus 10 interpolates the pixel value based on the direction of the edge determined by the luminance difference.

[0051] In addition, when the minimum gradient value minGrad is the color difference data gradZ or the color difference data gradN, the presence of an edge in the diagonal direction is determined by the color difference. However, the image processing apparatus 10 may not interpolate the pixel value based on the determined diagonal edge. This is because in the pixel array shown in FIG. 6, when an edge in the diagonal direction is detected, regardless of the direction, the pixel value of the target pixel B is interpolated using the pixel values of the four pixels B1 - B4 around the target pixel B.

[0052] When the pixel value interpolation process is performed using the pixel values of the pixels across the boundary portion of the image based on the detection of the diagonal edge, there is a possibility of false color generation. For this reason, when an edge in the diagonal direction is detected, the image processing apparatus 10 determines the direction of the edge based on the luminance difference, and interpolates the pixel value based on the determined direction of the edge.

[0053] For example, when the determined direction of the edge is the horizontal direction H, the image processing apparatus 10 interpolates the pixel value of the target pixel B with reference to the pixel values B2 and B3. When the determined direction of the edge is the vertical direction V, the image processing apparatus 10 interpolates the pixel value of the target pixel B with reference to the pixel values B1 and B4. In the example shown in FIG. 6, since the distances between the target pixel B and the reference pixels B2, B3 (or B1, B4) are equal to each other, the average of the pixel values of the pixels B2, B3 (or B1, B4) is set as the pixel value of the target pixel B. When the distances between the target pixel B and the plurality of reference pixels are different from each other, the average of the pixel values weighted according to the distances is set as the pixel value of the target pixel B.

[0054] In addition, when the image processing apparatus 10 interpolates pixel values based on the direction of an edge determined by luminance difference, it uses the same method as the interpolation of pixel values based on the direction of an edge determined by color difference. For example, when the direction of the edge determined by luminance difference is the horizontal direction H, the image processing apparatus 10 interpolates the pixel value of the target pixel B by referring to the pixel values of pixels B2 and B3. When the direction of the edge determined by luminance difference is the vertical direction V, the image processing apparatus 10 interpolates the pixel value of the target pixel B by referring to the pixel values of pixels B1 and B4. When the direction of the edge determined by luminance difference is the diagonal direction, or when the edge direction cannot be determined, the image processing apparatus 10 interpolates the pixel value of the target pixel B using the pixel values of the four surrounding pixels B1 - B4 of the target pixel B.

[0055] FIG. 7 shows an example of the flow of edge detection processing and pixel value interpolation processing performed by the image processing apparatus 10 of FIG. 3. That is, FIG. 7 shows an example of an image processing method by the image processing apparatus 10. The flow shown in FIG. 7 is realized, for example, when the CPU 20 of the image processing apparatus 10 executes an image processing program.

[0056] Detailed descriptions of the same processing as the above-described processing are omitted. Note that the flow shown in FIG. 7 may be realized by hardware such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) mounted on the image processing apparatus 10. Alternatively, the flow shown in FIG. 7 may be realized by cooperation of hardware and software.

[0057] First, in step S10, the image processing apparatus 10 generates luminance information for each pixel based on the image data acquired from the imaging apparatus 19. Next, in step S11, the image processing apparatus 10 uses the generated luminance information to detect an edge indicating the direction in which pixels with little change in luminance are arranged.

[0058] Next, in step S12, based on the image data acquired from the imaging device 19, the image processing apparatus 10 generates color difference information in each direction using the pixel values of the pixels around the target pixel that have the same color as the color of the target pixel after interpolation. Next, in step S13, the image processing apparatus 10 detects an edge indicating the direction in which pixels with little change in color difference are arranged using the generated color difference information.

[0059] Next, in step S14, when the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is equal to or greater than the threshold value T, the image processing apparatus 10 determines that an edge has been detected based on the color difference information and transfers the process to step S15. When the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is less than the predetermined threshold value T, the image processing apparatus 10 determines that an edge has not been detected based on the color difference information. In this case, since the image processing apparatus 10 performs interpolation processing using the edge detected based on the luminance information, the process is transferred to step S16.

[0060] In step S15, the image processing apparatus 10 determines whether to adopt the edge based on the color difference information detected in step S13 and transfers the process to step S17. In step S16, the image processing apparatus 10 determines whether to adopt the edge based on the luminance information detected in step S11 and transfers the process to step S17.

[0061] In step S17, the image processing apparatus 10 performs interpolation processing to convert the image data of the RGBIr pixel array into the image data of the Bayer array using the edge determined in step S15 or step S16, and ends the process shown in FIG. 7.

[0062] As described above, in this embodiment, the image processing apparatus 10 preferentially performs pixel value interpolation processing based on edges detected based on the difference in pixel values (color difference) over pixel value interpolation processing based on edges detected based on the luminance difference. Thereby, it is possible to suppress the occurrence of false colors due to the interpolation processing at the boundary of image regions with different colors, and it is possible to suppress the deterioration of the quality of the interpolated image. For example, when converting image data of an RGBIr pixel array into image data of a Bayer array, it is possible to suppress the occurrence of false colors in the converted image.

[0063] When the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is equal to or greater than the threshold value, the image processing apparatus 10 performs interpolation processing using the edge detected based on the color difference. In other words, when the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is less than the threshold value T, the image processing apparatus 10 does not perform interpolation processing using the edge detected based on the color difference, but performs interpolation processing using the edge detected based on the luminance. By performing the interpolation processing using the edge detected from the luminance when the difference in color difference is small and the accuracy of the edge detected from the color difference is low, it is possible to suppress the inclusion of noise such as false colors in the interpolated image due to the interpolation processing using the low-accuracy edge.

[0064] Before determining whether the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is equal to or greater than the threshold value T, the image processing apparatus 10 generates luminance information in advance, and uses the generated luminance information to detect an edge indicating a direction with little change in luminance. Thereby, even when the edge detected by the luminance is not used for the interpolation processing, it can be used for other image processing.

[0065] (Second Embodiment) FIG. 8 shows an example of the flow of edge detection processing and pixel value interpolation processing performed by the image processing apparatus in the second embodiment. That is, FIG. 8 shows an example of an image processing method by the image processing apparatus. Detailed description of the same processing as in FIG. 7 is omitted.

[0066] The image processing apparatus 10 that executes the flow shown in FIG. 8 is the same as the image processing apparatus 10 shown in FIGS. 1 to 3, and is mounted on the image processing system 100 together with the information processing apparatus 11 and the display apparatus 12. The flow shown in FIG. 8 is realized, for example, when the CPU 20 (FIG. 4) of the image processing apparatus 10 in FIG. 3 executes an image processing program.

[0067] Note that the flow shown in FIG. 8 may be realized by hardware such as an FPGA or an ASIC mounted on the image processing apparatus 10. Alternatively, the flow shown in FIG. 8 may be realized by cooperation between hardware and software.

[0068] The image processing system 100 is mounted on a moving body 200 such as an automobile, a transport robot, or a drone, for example. Note that the image processing system 100 may be a system that processes images acquired from an imaging device such as a surveillance camera, a digital still camera, or a digital camcorder.

[0069] First, in step S20, the image processing apparatus 10 generates color difference information in each direction based on the image data acquired from the imaging device 19, in the same manner as in step S12 of FIG. 7. Next, in step S21, if the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is equal to or greater than a predetermined threshold value T, the image processing apparatus 10 determines that an edge based on the color difference information has been detected, in the same manner as in step S14 of FIG. 7, and shifts the process to step S22. If the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is less than the predetermined threshold value T, the image processing apparatus 10 determines that no edge based on the color difference information has been detected, and thus determines to use the edge detected based on the luminance information, and shifts the process to step S23.

[0070] In step S22, the image processing apparatus 10 detects an edge indicating a direction in which the change in color difference is small using the color difference information generated in step S20, in the same manner as in step S13 of FIG. 7, and shifts the process to step S25. In step S23, the image processing apparatus 10 generates luminance information for each pixel based on the image data acquired from the imaging device 19, in the same manner as in step S10 of FIG. 7.

[0071] Next, in step S24, in the same manner as step S11 in FIG. 7, the image processing apparatus 10 uses the generated luminance information to detect an edge indicating a direction with little change in luminance, and shifts the process to step S25. In this way, the generation of luminance information and the detection of an edge based on the luminance information are performed only when the detection of an edge based on color difference information is not performed. Thereby, it is possible to suppress the wasteful execution of the processes in steps S23 and S24.

[0072] In step S25, the image processing apparatus 10 performs an interpolation process of converting the image data of the RGBIr pixel array into the image data of the Bayer array using the edge detected in step S22 or step S24, in the same manner as step S17 in FIG. 7. Then, the image processing apparatus 10 ends the process shown in FIG. 8.

[0073] As described above, also in this embodiment, the same effects as those of the above-described embodiments can be obtained. For example, by performing an interpolation process of the pixel value of the target pixel using the edge detected based on the color difference, it is possible to suppress the occurrence of false colors in the interpolated image at the boundary of the image regions having different colors.

[0074] Furthermore, in this embodiment, the image processing apparatus 10 generates luminance information only when the difference between the color difference in the vertical direction V and the color difference in the horizontal direction H is less than the threshold value T, and uses the generated luminance information to detect an edge indicating a direction with little change in luminance. Thereby, it is possible to suppress the wastefulness of the edge detection process by luminance when the edge detected by luminance is not used for other than the interpolation process.

[0075] As described above, the present invention has been described based on each embodiment, but the present invention is not limited to the requirements shown in the above embodiments. Regarding these points, it can be changed without departing from the gist of the present invention, and can be appropriately determined according to the application form.

Description of Reference Numerals

[0076] 10 Image processing device 10a Acquisition unit 10b First determination unit 10c Second determination unit 10d Interpolation processing unit 10e Output unit 11 Information processing device 12 Display device 13 ECU 14 Wireless communication device 15 Sensor 16 Drive device 17 Lamp device 18 Navigation device 19(19A, 19B, 19C, 19D, 19E) Imaging device 20 CPU 21 Interface device 22 Drive device 23 Auxiliary storage device 24 Memory device 30 Recording medium 100 Image processing system 200 Moving body BUS Bus H Horizontal direction IMG1, IMG2 Image data IMGS Image sensor PX Pixel V Vertical direction

Claims

1. An image processing apparatus that replaces pixel values received from an imaging device including a plurality of types of pixels having different wavelength regions of light to be detected with pixel values of pixels in a second pixel array different from the first pixel array of the imaging device, a first determination unit that determines the presence or absence of an edge indicating a direction in which pixels with little change in pixel value are arranged among peripheral pixels located around a target pixel to be interpolated and having the same color as the color of the target pixel after replacement; an interpolation processing unit that interpolates the pixel value of the target pixel using the pixel values of pixels along the direction of the edge detected based on the luminance of pixels located around the target pixel when it is determined that there is an edge, interpolates the pixel value of the target pixel using the pixel values of pixels along the direction of the edge detected based on the luminance of pixels located around the target pixel when it is determined that there is no edge, and interpolates the pixel value of the target pixel using all of the peripheral pixels located around the target pixel and having the same color as the color of the target pixel after replacement when no edge is detected based on the luminance of pixels located around the target pixel; An image processing apparatus having the above.

2. The first determination unit determines that there is an edge when the difference in pixel values of pixels arranged along a first direction with respect to the target pixel among the peripheral pixels and the difference in pixel values of pixels arranged along a direction intersecting the first direction among the peripheral pixels are equal to or greater than a predetermined threshold value The image processing apparatus according to Claim 1.

3. having a second determination unit that detects an edge based on the luminance of pixels located around the target pixel, The detection of the edge by the second determination unit is performed before the determination of the presence or absence of the edge by the first determination unit The image processing apparatus according to Claim 1 or Claim 2.

4. having a second determination unit that detects an edge based on the luminance of pixels located around the target pixel, The detection of the edge by the second determination unit is performed when it is determined by the first determination unit that there is no edge The image processing apparatus according to Claim 1 or Claim 2.

5. The first pixel array includes red, green, and blue pixels and pixels other than red, green, and blue, The interpolation processing unit converts the image data of the first pixel array including red, green, and blue pixels and the other pixels into image data of the second pixel array including red, green, and blue pixels and not including the other pixels The image processing apparatus according to any one of Claims 1 to 4.

6. The second pixel array is a Bayer array The image processing apparatus according to Claim 5.

7. The other pixels are pixels that detect light in the infrared wavelength region. The image processing apparatus according to claim 5 or claim 6.

8. An image processing method for replacing a pixel value of a target pixel with a pixel value of a pixel in a pixel array different from the pixel array of the imaging device based on pixel values received from an imaging device including a plurality of types of pixels having different wavelength regions of light to be detected, a first determination process for determining the presence or absence of an edge indicating a direction in which pixels with little change in pixel value are arranged among the peripheral pixels of the same color as the pixel after replacement, which are located around the target pixel; when it is determined that there is an edge, interpolating the pixel value of the target pixel using the pixel values of the pixels along the direction of the edge among the peripheral pixels, and when it is determined that there is no edge, using the pixel values of the pixels along the direction of the edge detected based on the luminance of the pixels located around the target pixel to interpolate the pixel value of the target pixel, and when no edge is detected based on the luminance of the pixels located around the target pixel, interpolating the pixel value of the target pixel using all of the peripheral pixels of the same color as the color of the target pixel after replacement, which are located around the target pixel; An image processing method for performing the above.

Images