Image processing apparatus and image processing method

Abstract

This image processing device comprises an extraction unit and a prediction unit. The extraction unit extracts, from image data indicating an image acquired by an image capture device and including a plurality of pixels, an unsaturated area in which pixel values are not saturated and a saturated area in which pixel values are saturated. The prediction unit predicts the pixel values of pixels of interest in the saturated area on the basis of the pixel values of boundary pixels of a boundary area in the unsaturated area relative to the saturated area. In this way, the data of an image in which a blown-out highlight has occurred can be used to predict the brightness value of a subject image in the area in which the blown-out highlight has occurred.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method.

Background Art

[0002] Recently, various techniques have been proposed to improve the recognizability of an image displayed on a display device by performing image processing on image data acquired by an imaging device mounted on a moving body such as a vehicle. For example, by synthesizing a plurality of image data acquired by a plurality of imaging devices or selecting any one of the plurality of image data, an image with suppressed white bleeding or black crushing can be made displayable.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, the pixel values of the image data in the area where blooming occurs reach the upper limit value that the imaging device can output. However, the actual luminance of the photographed area of the subject where blooming occurs is not uniform and gradually changes. Therefore, with a method of uniformly correcting the saturated area where the pixel value exceeds the upper limit value, a correct image of the subject cannot be obtained. Also, it is difficult to predict the subject image in the area where blooming occurs in one piece of image data acquired by one imaging device.

[0005] The present invention has been made in view of the above points, and an object thereof is to predict the luminance value of a subject in the area where blooming occurs using the image data in which blooming has occurred.

Means for Solving the Problems

[0006] In one aspect of the present invention, an image processing apparatus includes an extraction unit that extracts an unsaturated region where pixel values are not saturated and a saturated region where pixel values are saturated from image data indicating an image including a plurality of pixels acquired by an imaging device, and a prediction unit that predicts the pixel value of a target pixel in the saturated region based on the pixel value of a boundary pixel in a boundary region between the saturated region and the unsaturated region in the unsaturated region. Moreover, the image data is acquired for each pixel of a plurality of colors, the extraction unit extracts the unsaturated region and the saturated region from each of the image data of the plurality of colors, and the prediction unit performs a first prediction process of predicting the pixel value of the saturated region of the target color based on the ratio between the pixel value of the target color and the pixel value of another color that is not saturated in the boundary region of the target color, which is one of the plurality of colors. The ratio P1 / P2 between the pixel value P1 of the target color and the pixel value P2 of another color that is not saturated is changed so as to converge to the white balance value of the other color at any pixel position in the fully saturated region where all the pixel values of the plurality of colors are saturated. .

Effects of the Invention

[0007] According to the disclosed technology, it is possible to predict the luminance value of a subject image in the area where blooming occurs using the image data in which blooming has occurred.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description, image data may sometimes be simply referred to as an image.

[0010] (First Embodiment) FIG. 1 shows an example of an image processing system including an image processing apparatus according to the first embodiment. The image processing system 100 shown in FIG. 1 is mounted on a moving body 200 such as an automobile, for example. Imaging devices 19A, 19B, 19C, 19D, and 19E such as cameras are installed in front of, behind, on the left side, and on the right side of the moving body 200 in the traveling direction D, and in front of the interior of the moving body 200. Hereinafter, when the imaging devices 19A, 19B, 19C, 19D, and 19E are described without distinction, they are also referred to as the imaging device 19.

[0011] Note that the number and installation positions of the imaging devices 19 installed in the moving body 200 are not limited to those shown in FIG. 1. For example, one imaging device 19 may be installed only in front of the moving body 200, or two imaging devices 19 may be installed only in the front and the rear. Alternatively, the imaging device 19 may be installed on the ceiling of the moving body 200. Further, the moving body 200 on which the image processing system 100 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory. Further, the image processing system 100 may be a system that processes images acquired from imaging devices other than the imaging devices installed in the moving body 200, such as a surveillance camera, a digital still camera, or a digital camcorder.

[0012] The image processing system 100 includes an image processing apparatus 10, an information processing apparatus 11, and a display apparatus 12. Note that in FIG. 1, for easy understanding of the description, the image processing system 100 is superimposed on an image diagram of the moving body 200 viewed from above. However, in reality, the image processing apparatus 10 and the information processing apparatus 11 are mounted on a control board or the like mounted on the moving body 200, and the display apparatus 12 is installed at a position visible to a person inside the moving body 200. Note that the image processing apparatus 10 may be mounted on a control board or the like as a part of the information processing apparatus 11. The image processing apparatus 10 is connected to each imaging device 19 via a signal line or wirelessly.

[0013] FIG. 2 shows an example of the functional configuration of the image processing apparatus 10 in FIG. 1. The image processing apparatus 10 includes an acquisition unit 10a, an extraction unit 10b, a prediction unit 10c, and an output unit 10d. The acquisition unit 10a acquires image data indicating an image around the moving body 200 captured by each imaging device 19. The extraction unit 10b and the prediction unit 10c perform image processing (correction processing) on the image data acquired from each imaging device 19. The output unit 10d outputs the result of the image processing to at least one of the display device 12 and the information processing device 11.

[0014] Returning to FIG. 1, the display device 12 is, for example, a display of a side mirror monitor, a rear mirror monitor, or a navigation device installed in the moving body 200. Note that the display device 12 may be a display provided on a dashboard or the like, or a head-up display (HUD) that projects an image onto a projection plate or a windshield or the like.

[0015] The information processing device 11 includes a computer such as a processor that performs recognition processing and the like based on the image data received via the image processing device 10. For example, the information processing device 11 mounted on the moving body 200 detects other moving bodies, signals, signs, white lines on the road, and people by performing recognition processing on the image data, and determines the situation around the moving body 200 based on the detection results. Note that the information processing device 11 may include an automatic driving control device that controls the movement, stop, right turn, and left turn of the moving body 200.

[0016] FIG. 3 shows an outline of the configuration of various devices mounted on the moving body 200 in FIG. 1. The moving body 200 includes an image processing device 10, an information processing device 11, a display device 12, at least one ECU (Electronic Control Unit) 13, and a wireless communication device 14 that are interconnected via an internal network. The moving body 200 also includes a sensor 15, a driving device 16, a lamp device 17, a navigation device 18, and an imaging device 19. For example, the internal network is a vehicle-mounted network such as a CAN (Controller Area Network) or Ethernet (registered trademark).

[0017] The image processing device 10 corrects the image data (frame data) acquired by the imaging device 19 and generates corrected image data. The image processing device 10 may record the generated corrected image data in an external or internal recording device.

[0018] The information processing device 11 may function as a computer that controls each part of the moving body 200. The information processing device 11 controls the entire moving body 200 by controlling the ECU 13. The information processing device 11 may recognize an object outside the moving body 200 based on the image generated by the image processing device 10, and may track the recognized object.

[0019] The display device 12 displays the image and corrected image generated by the image processing device 10. When the moving body 200 moves backward (backs up), the display device 12 may display the image in the backward direction of the moving body 200 in real time. Further, the display device 12 may display the image output from the navigation device 18.

[0020] The ECUs 13 are respectively provided corresponding to mechanism parts such as an engine or a transmission. Each ECU 13 controls the corresponding mechanism part based on an instruction from the information processing device 11. The wireless communication device 14 communicates with devices outside the moving body 200. The sensor 15 is a sensor that detects various types of information. The sensor 15 may include, for example, a position sensor that acquires the current position information of the moving body 200. Further, the sensor 15 may include a speed sensor that detects the speed of the moving body 200.

[0021] The drive device 16 is various devices for moving the moving body 200. The drive device 16 may include, for example, an engine, a steering device (steering), and a braking device (brake), etc. The lamp device 17 is various lamps mounted on the moving body 200. The lamp device 17 may include, for example, a headlamp (headlamp, headlight), a lamp of a direction indicator (winker), a backlight, and a brake lamp, etc. The navigation device 18 is a device that guides the route to the destination by voice and display.

[0022] Figure 4 shows an example of the configurations of the image processing device 10 and the information processing device 11 in FIG. 3. Since the configurations of the image processing device 10 and the information processing device 11 are similar to each other, the configuration of the image processing device 10 will be described below. For example, the image processing device 10 has a CPU 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a memory device 24 that are interconnected by a bus BUS.

[0023] The CPU 20 executes various image processes described later by executing the image processing program stored in the memory device 24. The interface device 21 is used to connect to a network (not shown). The auxiliary storage device 23 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc., and holds an image processing program, image data, and various parameters used for image processing, etc.

[0024] The memory device 24 is, for example, a DRAM (Dynamic Random Access Memory), etc., and holds the image processing program, etc. transferred from the auxiliary storage device 23. The drive device 22 has an interface for connecting the recording medium 30, and, for example, transfers the image processing program stored in the recording medium 30 to the auxiliary storage device 23 based on an instruction from the CPU 20. Note that the drive device 22 may transfer the image data, etc. stored in the auxiliary storage device 23 to the recording medium 30.

[0025] FIG. 5 shows an example of image processing of the image IMG acquired by the imaging device 19 of FIG. 1. For example, the imaging device 19 is an image sensor in which pixels including three photoelectric conversion elements that detect red (R), green (G), and blue (B) light are arranged in a matrix. Hereinafter, each of the three photoelectric conversion elements is also referred to as a pixel. The pixel values of the red (R), green (G), and blue (B) pixels are also referred to as R values, B values, and G values, respectively.

[0026] Note that the photoelectric conversion elements included in the pixels of the imaging device 19 are not limited to red (R), green (G), and blue (B). For example, the pixels of the imaging device 19 may include a photoelectric conversion element that detects near-infrared light. Alternatively, the pixels of the imaging device 19 may include photoelectric conversion elements for cyan, yellow, and magenta, or may include photoelectric conversion elements for cyan, yellow, and red.

[0027] The arrow of the dashed line in the image indicates an image in which the imaging device 19 acquires image data of the subject by raster scan. Note that the image IMG may be acquired by any one of the other imaging devices 19A - 19D, or may be a composite image obtained by combining images acquired by a plurality of imaging devices 19.

[0028] The image IMG in FIG. 5 includes, for example, the sun S as a subject image with high luminance. Therefore, the luminance is high in the pixels close to the sun S. The image IMG in FIG. 5 is classified into four sections with a curve indicated by a dashed line as a boundary.

[0029] In the first section farthest from the sun S, the R value, G value, and B value are equal to or less than the maximum pixel value (upper limit value) that can be acquired by the imaging device 19 (no saturation). That is, the first section is an unsaturated region where the R value, G value, and B value are not saturated.

[0030] In the second section next farthest from the sun S, the R value and B value are equal to or less than the upper limit value, and the G value is greater than the upper limit value (G saturation). That is, the second section is an unsaturated region where the R value and B value are not saturated, and a saturated region where the G value is saturated.

[0031] In the third section, which is closer to the sun S than the second section, the B value is below the upper limit value, and the G value and R value are greater than the upper limit value (GR saturation). That is, the third section is an unsaturated region where the B value is not saturated, and a saturated region where the G value and R value are saturated respectively.

[0032] In the fourth section, which is closest to the sun S, the G value, R value, and B value are greater than the upper limit value (GRB saturation). That is, the fourth section is a fully saturated region where all of the G value, R value, and B value are saturated. When the pixel value is saturated, the image acquired by the imaging device 19 cannot correctly represent the color of the subject. An image in which all of the G value, R value, and B value are saturated becomes white, and an image called so-called white blooming is generated.

[0033] The image processing device 10 performs an extraction process of pixel values using the image data acquired for each pixel of a plurality of colors of the imaging device 19, and extracts an unsaturated region and a saturated region for each of the R value, G value, and B value. The image processing device 10 performs at least one of a first prediction process of predicting a pixel value using a ratio between an unsaturated self-color component and a different-color component, and a second prediction process of predicting a pixel value using an unsaturated self-color component.

[0034] In the first prediction process, the image processing device 10 predicts the G value of the second section from the R value of the second section, for example, based on the ratio between the G value and the R value in the first section where the G value and the R value are not saturated. Further, the image processing device 10 predicts the G value of the second section from the B value of the third section, for example, based on the ratio between the G value and the B value in the first section where the G value and the B value are not saturated.

[0035] The image processing device 10 predicts the R value of the fourth section from the predicted value of the R value of the fourth section, for example, based on the predicted B value of the fourth section predicted in advance and the ratio between the R value and the B value in a section where the R value and the B value are not saturated. Further, the image processing device 10 predicts the G value of the fourth section from the predicted value of the B value of the fourth section, for example, based on the ratio between the G value and the B value in the first section where the G value and the B value are not saturated.

[0036] In the second prediction process, for example, based on the pixel values of a plurality of G-color boundary pixels located in the boundary region with the second section (saturation region) in the first section where the G value is not saturated, the image processing apparatus 10 predicts the pixel values in the saturation region (from the second section to the fourth section) of the G value. Further, for example, based on the pixel values of a plurality of R-color boundary pixels located in the boundary region with the third section (saturation region) in the second section where the R value is not saturated, the image processing apparatus 10 predicts the pixel values in the saturation region (from the third section to the fourth section) of the R value.

[0037] Furthermore, for example, based on the pixel values of a plurality of B-color boundary pixels located in the boundary region with the fourth section (saturation region) in the third section where the B value is not saturated, the image processing apparatus 10 predicts the pixel values in the saturation region (the fourth section) of the B value. Specific examples of the first prediction process and the second prediction process are described with reference to FIGS. 8 to 10.

[0038] By performing at least one of the first prediction process and the second prediction process, the image processing apparatus 10 can calculate, as a predicted value, a pixel value corresponding to the actual luminance value of the subject even when receiving a saturated pixel value from the imaging apparatus 19. Thereby, the image processing apparatus 10 can correct the image data including the pixel values in the region where blooming has occurred into image data without blooming.

[0039] Note that, in FIGS. 5 and 6 and subsequent figures, an example in which the pixel values of the G value, the R value, and the B value are saturated in this order is described. This is because, in general, in an RGB image sensor, the sensitivities of the G component, the R component, and the B component are high in this order. However, the order of the colors in which the pixel values are saturated is not limited to the example shown in FIG. 5.

[0040] FIG. 6 shows the pixel values that the imaging apparatus 19 can output and the pixel values that the imaging apparatus 19 cannot output in the image data of the subject acquired by the imaging apparatus 19 in FIG. 3. The first section to the fourth section shown in FIG. 6 respectively correspond to the first section to the fourth section in FIG. 5. For example, FIG. 6 shows the G value, the R value, and the B value in the raster scan (5) of FIG. 5. The horizontal axis in FIG. 6 indicates the position (horizontal coordinate) of the pixel on the raster scan (5).

[0041] The vertical axis in FIG. 6 shows, on a logarithmic scale, the ratio of pixel values when the maximum pixel value that can be acquired by the imaging device 19 is set to "1". For example, when the imaging device 19 outputs pixel values in 8 bits, "1" on the vertical axis corresponds to "255" of the pixel value. When the luminance of the subject exceeds the maximum pixel value ("1" on the vertical axis), the imaging device 19 outputs that maximum pixel value ("1" on the vertical axis). Hereinafter, the maximum pixel value is also referred to as the upper limit value.

[0042] In FIG. 6, the G value, R value, and B value indicated by the solid line are the pixel values output by the imaging device 19. The G value, R value, and B value indicated by the dashed line are the pixel values corresponding to the actual luminance of the subject. The image processing device 10 predicts pixel values exceeding the upper limit value indicated by the dashed line by performing at least one of the first prediction process and the second prediction process.

[0043] FIG. 7 shows an example of a method for predicting image values performed by the image processing device 10 in FIG. 3. That is, FIG. 7 shows an example of an image processing method by the image processing device 10. The flow shown in FIG. 7 is realized, for example, when the CPU 20 of the image processing device 10 executes an image processing program. A specific example of the image processing method performed by the image processing device 10 will be described after FIG. 8.

[0044] Note that the flow shown in FIG. 7 may be realized by hardware such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) mounted on the image processing device 10. Alternatively, the flow shown in FIG. 7 may be realized by cooperation between hardware and software.

[0045] For example, one loop of the process shown in FIG. 7 is executed for each pixel of the raster scan lines shown in FIG. 5. In the image IMG shown in FIG. 5, when the sun S is at the upper left corner of the image IMG, the first to fourth sections are arranged from right to left. Also, in the image IMG shown in FIG. 5, when the sun S is near the center of the image IMG, there are the first to fourth sections arranged from left to right in the image IMG and the first to fourth sections arranged from right to left in the image IMG.

[0046] First, in step S10, when the image processing apparatus 10 detects the boundary between the first section and the second section based on at least any one of the G value, the R value, and the B value, steps S12 and S20 are executed. When the image processing apparatus 10 does not detect the boundary between the first section and the second section, step S40 is executed.

[0047] In step S12, the image processing apparatus 10 creates an approximation formula by calculating the parameters of the approximation formula for calculating the pixel value that exceeds the upper limit value in the second section. Next, in step S14, the image processing apparatus 10 executes image processing for calculating the pixel value that exceeds the upper limit value using the approximation formula set with the calculated parameters. After this, the process proceeds to step S40.

[0048] In step S20, when the image processing apparatus 10 detects the boundary between the second section and the third section based on at least any one of the G value, the R value, and the B value, steps S22 and S30 are executed. When the image processing apparatus 10 does not detect the boundary between the second section and the third section, step S40 is executed.

[0049] In step S22, the image processing apparatus 10 creates an approximation formula by calculating the parameters of the approximation formula for calculating the pixel value that exceeds the upper limit value in the third section. Next, in step S24, the image processing apparatus 10 executes image processing for calculating the pixel value that exceeds the upper limit value using the approximation formula set with the calculated parameters. After this, the process proceeds to step S40.

[0050] In step S30, when the image processing apparatus 10 detects the boundary between the third section and the fourth section based on at least one of the G value, the R value, and the B value, it executes step S32. When the image processing apparatus 10 does not detect the boundary between the third section and the fourth section, it executes step S40.

[0051] In step S32, the image processing apparatus 10 creates an approximation formula by calculating the parameters of the approximation formula for calculating the pixel value exceeding the upper limit value in the fourth section. Next, in step S34, the image processing apparatus 10 executes image processing for calculating the pixel value exceeding the upper limit value using the approximation formula set with the calculated parameters. After this, the process proceeds to step S40.

[0052] In step S40, when the image processing apparatus 10 continues the image processing, it returns to step S10. When it does not continue the image processing, for example, when the processing of one screen is completed, it ends the processing shown in FIG. 7. Note that the image processing apparatus 10 may execute image processing such as noise removal processing or edge enhancement processing after step S40 and before returning to step S10.

[0053] FIG. 8 shows an example of the approximation formula used in the image processing of FIG. 7. In the formula for each section, the symbol ^ attached above the symbols G, R, and B indicates the predicted pixel value. Hereinafter, the predicted pixel value is represented by pixel values G^, R^, and B^. The dotted underline shows an example of the approximation formula used for prediction by different-color components in the first prediction process. The dashed-dotted underline shows an example of the approximation formula used for prediction by non-saturated same-color components in the second prediction process. An example of predicting the pixel value by the first prediction process is shown in FIG. 9, and an example of predicting the pixel value by the second prediction process is shown in FIG. 10.

[0054] The symbol α represents the application ratio of the first prediction process, and the symbol 1-α represents the application ratio of the second prediction process. That is, the symbol α represents the blending ratio for mixing the pixel value predicted by the first prediction process and the pixel value predicted by the second prediction process at a predetermined ratio. When α = 1, only the first prediction process (prediction by different color components) is applied, and when α = 0, only the second prediction process (prediction by non-saturated self-color components) is applied.

[0055] By applying the blending ratio α, it becomes possible to appropriately blend the predicted pixel value by the first prediction process and the predicted pixel value by the second prediction process in accordance with the characteristics of the subject (image data), Saturated region and improve the accuracy of the prediction of the pixel value of.

[0056] In each formula, R represents the pixel value of the R component, and B represents the pixel value of the B component. The coefficient a R represents the coefficient for calculating the pixel value R^ from the pixel value R or the pixel value R^. The coefficient a B represents the coefficient for calculating the pixel value G^ or the pixel value R^ from the pixel value B or the pixel value B^. For example, the coefficient a R , a B and the blending ratio α are parameters set in each approximation formula for calculating the pixel values G^, R^, B^.

[0057] The coefficient a R represents the predicted GR ratio as shown in formula (5-1). The predicted GR ratio is the ratio G value / R value (P1 / P2) of the G value (P1) of the target color G in the first interval and the R value (P2) of another color R. The coefficient a B represents the predicted GB ratio as shown in formula (5-2). The predicted GB ratio is the ratio G value / B value (P1 / P2) of the G value (P1) of the target color G and the B value (P2) of another color B. The calculation of the predicted GR ratio and the predicted GB ratio is preferably performed at the boundary with the second interval where the difference in pixel values is the largest in the first interval.

[0058] The symbol β indicates the application ratio of Expression (5-1) and Expression (5-2). That is, the symbol β indicates the priority (weight) of the predicted GR ratio and the predicted GB ratio. When β = 1, only the predicted GR ratio is applied. When β = 0, only the predicted GB ratio is applied. Note that since the difference between the pixel values G and B is larger than the difference between the pixel values G and R, the use of the predicted GB ratio tends to result in higher prediction accuracy of the pixel value Ĝ. For this reason, it is preferable to decrease β and make the weight of the predicted GB ratio larger than the weight of the predicted GR ratio.

[0059] In this way, by applying the priority ratio β, it becomes possible to appropriately set whether to preferentially use the pixel values R or B according to the characteristics of the subject (image data), Saturated region and improve the accuracy of prediction of the pixel value.

[0060] Coefficient a C is not used in this embodiment, and thus is set to "0" as shown in Expression (5-3). Note that the predicted RB ratio, which is the ratio R value / B value between the R value and the B value in the first interval, may be used in the approximate expression.

[0061] In the fourth interval where the pixel values of all colors are saturated, the first prediction process cannot be applied until one pixel value is predicted. For this reason, in the fourth interval, the image processing apparatus 10 first calculates the pixel value B̂ using the second prediction process as shown in Expression (4-1), and calculates the pixel values R̂ and Ĝ using the calculated pixel value B̂.

[0062] Expression (4-1) is obtained by replacing the symbol G in Expression (6-2) with the symbol B. Expressions (6-1) and (6-2) show an example of predicting the G value in the saturation interval using the component of the G value. In Expressions (4-1), (6-1), (6-2), and (6-3), the symbol x indicates the coordinate in the horizontal direction of the image (the raster scan direction in FIG. 5), and the symbol y indicates the coordinate in the vertical direction of the image.

[0063] Expression (6-1) is the pixel value G(t x , y) of the boundary pixel adjacent to the saturation interval of the pixel value and the pixel value G(t xThe difference from (-1, y) is defined as the differential value of the pixel value. The boundary pixel is included in the non-saturated section (for example, the first section in FIG. 6). The symbol t x represents the horizontal position (coordinate) of the boundary pixel, and the symbol t x -1 represents the horizontal position (coordinate) of the pixel immediately before one of the pixels adjacent to the saturated region at the boundary pixel.

[0064] The first term of Equation (6-2) represents the pixel value G(t x , y) of the boundary pixel adjacent to the saturated section. The second term of Equation (6-2) represents the product of the horizontal distance between the prediction pixel x for predicting the pixel value and the boundary pixel t x and the differential value obtained by Equation (6-1). That is, the second term of Equation (6-2) represents the change amount (predicted value) of the pixel value from the boundary pixel t x to the prediction pixel x.

[0065] Equations (6-1) and (6-2) show an example of linearly predicting the pixel value by first-order differentiation. However, as shown in Equation (6-3), by using second-order differentiation, the pixel value may be approximated by a second-order function. Further, it may be extended up to an nth-order function. In this case, it is equivalent to the process of performing a Taylor expansion on the change in the pixel value of the boundary pixel t x .

[0066] FIG. 9 shows an example of predicting the pixel value by the first prediction process described in FIG. 8. In the first prediction process, the image processing apparatus 10 predicts the pixel value of the white color using the different-color components. In FIG. 9, the coefficient α in FIG. 8 is set to "0". In the first section where not all pixels of all colors are saturated, the image processing apparatus 10 calculates the predicted GR ratio (G value / R value) and the predicted GB ratio (G value / B value) using the G value, R value, and B value.

[0067] In the second section, the image processing apparatus 10 calculates the pixel value G^ by adding the product of the coefficient a R and the pixel value R and the product of the coefficient a B and the pixel value B at each pixel position using the first term of Equation (2) in FIG. 8.

[0068] In the third interval, the image processing apparatus 10 first uses the first term of Equation (3-1) in FIG. 8 at each pixel position to calculate the pixel value R^ by multiplying the coefficient a B and the pixel value B. Next, the image processing apparatus 10 uses the first term of Equation (3-2) in FIG. 8 to calculate the pixel value G^ by adding the product of the coefficient a R and the pixel value R^ and the product of the coefficient a B and the pixel value B.

[0069] In the fourth interval, the image processing apparatus 10 calculates the pixel value B^ using Equation (4-1) in FIG. 8 at each pixel position. Next, the image processing apparatus 10 uses the first term of Equation (4-2) in FIG. 8 to calculate the pixel value R^ by multiplying the coefficient a B and the pixel value B^. Next, the image processing apparatus 10 uses the first term of Equation (4-3) in FIG. 8 to calculate the pixel value G^ by adding the product of the coefficient a R and the pixel value R^ and the product of the coefficient a B and the pixel value B^. In this way, by the first prediction process, the pixel values in the saturated region can be predicted using the ratio of pixel values with different colors in the non-saturated region.

[0070] As described above, in the first prediction process, the parameters of the approximate formula are obtained. Note that the predicted pixel values G^, R^, and B^ by the image processing apparatus 10 may have a lower prediction accuracy as they are farther from the non-saturated interval. For this reason, for example, the image processing apparatus 10 corrects the predicted GR ratio and the predicted GB ratio so that they approach achromaticity as they are farther from the non-saturated interval. The lower part of FIG. 9 shows the correction curves of the predicted GR ratio and the predicted GB ratio.

[0071] The correction curve of the predicted GR ratio is set such that the predicted GR ratio becomes the white balance value (WB) of the R component at a predetermined pixel position in the fourth interval where all RGB colors are saturated. The correction curve of the predicted GB ratio is set such that the predicted GB ratio becomes the white balance value of the B component at a predetermined pixel position in the fourth interval. For example, since the white balance calculation of the B value is the product of the B value and the white balance value of the B component, after the white balance correction, G value = B value (achromatic color).

[0072] Thus, even in the fourth section where all of the G value, R value, and B value are saturated, for example, it is possible to prevent the color of the corrected image of the pixel value from deviating from the color of the actual subject. As a result, when the image data including the pixel values G^, R^, and B^ predicted by the first prediction process is compressed and displayed on the display device 12 as described with reference to FIG. 11 to be described later, an image without a sense of incongruity can be displayed.

[0073] Note that the G value, R value, and B value output from the imaging device 19 are fixed to the upper limit value (for example, value 255) after saturation. For this reason, the actual GR ratio gradually decreases in the second section and becomes 1.0 at the end point of the second section. The actual GB ratio gradually decreases in the second section and the third section and becomes 1.0 at the end point of the third section.

[0074] FIG. 10 shows an example of predicting a pixel value by the second prediction process described in FIG. 8. In the second prediction process, the image processing apparatus 10 predicts the pixel value of the self-color using the self-color component in the non-saturated section. In FIG. 10, the method of predicting the pixel value G^ is shown, but the methods of predicting the pixel values R^ and B^ are the same as the method of predicting the pixel value G^.

[0075] First, the image processing apparatus 10 calculates a differential value dG / dx, which is the slope of the change in the G value at the boundary (saturation boundary) between the first section and the second section where the G value is saturated, as shown in Expression (6-1) of FIG. 8. Next, the image processing apparatus 10 predicts the pixel value G^ at an arbitrary pixel position x in the second section, the third section, and the fourth section by applying the calculated differential value dG / dx to Expression (6-2) of FIG. 8. In this way, by the second prediction process, the pixel value can be predicted using the change rate of the pixel value of the self-color component.

[0076] FIG. 11 shows an example of a method for compressing pixel values predicted using at least one of the first prediction process and the second prediction process. If the predicted pixel value exceeds the upper limit of the pixel values that can be displayed on the display device 12 installed in the moving body 200, the image displayed on the display device 12 will have white streaks. For this reason, the image processing apparatus 10 has a function of compressing the predicted pixel values. In FIG. 11, it is assumed that the maximum value of the pixel values that can be displayed on the display device 12 is "255", and the maximum value of the predicted pixel values is "512".

[0077] The image processing apparatus 10 uses a predetermined conversion formula according to the maximum value of the predicted pixel values to convert the predicted pixel values into pixel values that can be displayed on the display device 12. That is, after predicting the pixel values in the saturation region in the image data, the image processing apparatus 10 compresses the pixel values of each pixel in the image data so that the maximum value of the predicted pixel values falls within the pixel values in the non-saturation region.

[0078] Then, the image processing apparatus 10 outputs the converted pixel values to the display device 12 as output pixel values. Thereby, even when the predicted pixel value exceeds the maximum pixel value that can be displayed on the display device 12, an image with suppressed white streaks can be displayed on the display device 12.

[0079] Note that the conversion formula for converting the predicted pixel values into output pixel values may change the parameters in the conversion formula according to the maximum value of the predicted pixel values. In order to prevent the hue of the image displayed on the display device 12 from changing, for example, when the pixel value G^ is compressed, it is preferable that the non-saturated pixel values R and B are also compressed.

[0080] The image processing apparatus 10 may arbitrarily set the range for compressing pixel values in an image of one screen. For example, in FIG. 5, the image processing apparatus 10 may compress the pixel values only in the empty regions in the image. Also, the image processing apparatus 10 may divide the image into partial images of a predetermined size and compress the pixel values for the partial images that include pixels with saturated pixel values. Note that the image processing apparatus 10 may suppress the unnatural change in the pixel values from being visible by performing smoothing processing on the boundary between the region to be compressed and the region not to be compressed.

[0081] In addition, when image recognition processing or the like is performed by the information processing apparatus 11 in FIG. 3 using the image data including the predicted pixel values, the image processing apparatus 10 may output the image data including the predicted pixel values to the information processing apparatus 11 without compressing it. In this case, the image processing apparatus 10 may output the image data including the predicted pixel values to the information processing apparatus 11 via the auxiliary storage device 23 or the memory device 24 in FIG. 4.

[0082] As described above, in this embodiment, even when the pixel value of the subject acquired by the imaging device 19 exceeds the upper limit value of the pixel values that can be acquired by the imaging device 19, the correct pixel value can be predicted. For example, the image processing apparatus 10 can predict the pixel value in the saturated region by using the ratio of pixel values having different colors in the non-saturated region by the first prediction process. Further, the image processing apparatus 10 can predict the pixel value in the saturated region by using the change rate of the pixel value of the self-color component by the second prediction process.

[0083] Thereby, even when the image processing apparatus 10 receives the pixel value in the saturated state from the imaging device 19, it can calculate the pixel value corresponding to the actual luminance value of the subject as the predicted value. As a result, the image processing apparatus 10 can correct the image data including the pixel values in the area where whiteout has occurred into the image data without whiteout, and perform highly accurate image recognition processing using the corrected image data.

[0084] In addition, by compressing the pixel values of the image data including the corrected pixel values, it is possible to display an image of a subject without whiteout on the display device 12 even when the luminance of the subject is high. As a result, the quality of the image displayed on the display device 12 can be improved.

[0085] By applying the blending ratio α, the image processing apparatus 10 can appropriately blend the predicted pixel value by the first prediction process and the predicted pixel value by the second prediction process according to the characteristics of the subject (image data). For example, by changing the blending ratio α according to the environment (weather, degree of sunlight, time, etc.) in which the image processing apparatus 10 is used, Saturated regionIt becomes possible to improve the accuracy of predicting the pixel value. Also, by changing the blending ratio α according to the characteristics of the subject image, Saturated region it becomes possible to improve the accuracy of predicting the pixel value.

[0086] In the first prediction process, by converging the corrected pixel value in the fourth section to the white balance value, it is possible to prevent the color of the image after correcting the pixel value from deviating from the color of the actual subject. As a result, when compressing the pixel values of the image data including the corrected pixel values and displaying them on the display device 12, an image without a sense of incongruity can be displayed.

[0087] (Second Embodiment) FIG. 12 shows an example of image processing performed by the image processing apparatus according to the second embodiment. The image processing apparatus 10 and the image processing system 100 including the image processing apparatus 10 of this embodiment have the same configuration as that in FIGS. 1 to 4 and may be mounted on the moving body 200.

[0088] In the first embodiment, the image processing in the case where the first section to the fourth section are sequentially arranged from the left side to the right side of the image IMG has been described. However, in an actual subject, as shown in FIG. 12, the positions, sizes, shapes, etc. of the first section to the fourth section are various. Also, there are multiple directions of the straight line crossing the section boundary.

[0089] Therefore, in this embodiment, the image processing apparatus 10 predicts the pixel values that exceed the upper limit value in each of the multiple scanning directions crossing the section boundary. The image processing apparatus 10 acquires the distribution of the first section to the fourth section in the image IMG (the annular broken line in FIG. 12) while imaging the subject in order to determine the section boundary. By acquiring the distribution, the image processing apparatus 10 can detect the pixels whose pixel values exceed the upper limit value for each color.

[0090] Then, for each pixel exceeding the upper limit value included in the second to fourth intervals, the image processing apparatus 10 determines a plurality of arrangement directions (scanning directions) used for predicting the pixel value. In the example shown in FIG. 12, eight scanning directions toward the target pixel indicated by the black circle included in the fourth interval are shown, but the number of scanning directions is not limited to eight.

[0091] Next, for each scanning direction, the image processing apparatus 10 calculates the parameters of the approximation formula for each pixel using at least one of the first prediction process (prediction by different color components) and the second prediction process (prediction by non-saturated self-color components) described in the first embodiment. The image processing apparatus 10 predicts the pixel value for each color exceeding the upper limit value using the parameters calculated in each scanning direction.

[0092] Then, the image processing apparatus 10 calculates the predicted pixel value of the target pixel by weighted-averaging the pixel values at the target pixels for each color predicted in a plurality of scanning directions according to the distance to the saturation boundary, which is the boundary between the non-saturated region and the saturated region. In FIG. 12, one target pixel is shown in the fourth interval, but the image processing apparatus 10 sequentially sets all the pixels included in the second, third, and fourth intervals as the target pixels, and acquires the pixel values of the colors exceeding the upper limit value at each pixel. Then, the image processing apparatus 10 predicts the pixel values in a plurality of scanning directions for each target pixel and calculates the weighted average. Note that the image processing apparatus 10 may compress the acquired predicted pixel values as described in FIG. 11.

[0093] As described above, also in this embodiment, similar to the above-described embodiments, even when the pixel value of the subject acquired by the imaging apparatus 19 exceeds the upper limit value of the pixel values that can be acquired by the imaging apparatus 19, the correct pixel value can be predicted. Further, in this embodiment, the correct pixel value can be predicted regardless of the position of the subject with high luminance. At this time, by weighted-averaging the pixel values predicted in a plurality of scanning directions according to the distance to the saturation boundary, the accuracy of the predicted pixel value can be improved compared to the case where the pixel value is predicted in a single scanning direction.

[0094] (Third Embodiment) FIG. 13 shows an example of image processing performed by the image processing apparatus according to the third embodiment. Detailed descriptions of elements similar to those in the above-described embodiments are omitted. The image processing apparatus 10 and the image processing system 100 including the image processing apparatus 10 in this embodiment have the same configuration as that in FIGS. 1 to 4 and may be mounted on the moving body 200.

[0095] In the first and second embodiments, the scanning direction for predicting the pixel value is one-dimensional (linear). In this case, there is a possibility that artifacts (noise) on the line may occur depending on the scanning direction. Therefore, in this embodiment, the image processing apparatus 10 does not perform processing that depends on the scanning direction, but performs a filter operation on the peripheral pixels around the target pixel for which the pixel value is predicted. Thereby, it is possible to eliminate the processing that depends on the scanning direction and suppress the occurrence of artifacts.

[0096] Note that FIG. 13 shows an example of obtaining the pixel value of the target pixel included in the second section. However, the image processing apparatus 10 sequentially sets all the pixels included in the second section, the third section, and the fourth section as the target pixels, and predicts the pixel value of the color that exceeds the upper limit value for each pixel. Further, as described with reference to FIG. 11, the image processing apparatus 10 may compress the obtained predicted pixel values.

[0097] Reference numerals tx and ty in FIG. 13 indicate the range of the peripheral pixel region including the pixels used for predicting the pixel value of the target pixel. The image processing apparatus 10 performs an extraction process of extracting a boundary region from the peripheral pixel region including the target pixel. Here, the boundary region is a region including a predetermined number of pixels (boundary pixels) along the direction facing the unsaturated region and adjacent to the boundary with the saturated region in the unsaturated region of the pixels having the same color as the target pixel. Hereinafter, the pixel value of the boundary pixel is also referred to as the boundary pixel value.

[0098] The image processing apparatus 10 predicts the pixel value of a target pixel by using the pixel values of boundary pixels included in a peripheral pixel region. For example, in FIG. 13, in the first interval, the pixel value of the target pixel of green G in the second interval is predicted by using the pixel values of boundary pixels in a boundary region adjacent to the boundary of the second interval where the G value is in the unsaturated region.

[0099] Then, the image processing apparatus 10 calculates, for each pixel, parameters of an approximate expression by using at least any one of a first prediction process (prediction by different-color components) and a second prediction process (prediction by non-saturated self-color components) based on a plurality of boundary pixel values. The image processing apparatus 10 predicts the pixel values for each color that exceed the upper limit value by using the calculated parameters.

[0100] Furthermore, the image processing apparatus 10 performs weighted averaging on the predicted pixel values for each color according to the distance from the boundary pixel to the target pixel, and determines the predicted pixel value of the color that exceeds the upper limit value of the target pixel. Examples of calculating the predicted pixel values by the first prediction process and the second prediction process are described with reference to FIGS. 14 and 15.

[0101] FIG. 14 shows an example of an approximate expression in the case where the prediction of the pixel values shown in FIG. 13 is performed by using the first prediction process (prediction by different-color components). FIG. 14 shows an example of calculating the predicted pixel value of pixel G, but the predicted pixel values of pixel R and pixel B can be calculated in the same manner.

[0102] For example, similar to the first term of Equation (2) and Equations (5-1), (5-2), and (5-3) in FIG. 8, the coefficients a R , a B can be predicted from the non-saturated pixels around the target pixel. As described above, the coefficient a R is calculated as the ratio of G to R, and the coefficient a B is calculated as the ratio of G to B. Since there are a plurality of non-saturated pixels, it is preferable to obtain a weighted average according to the distance.

[0103] The function V(tx, ty) shown in Equation (7-1) is used in term C of the approximate formula (7-2) for calculating the predicted GR ratio. The function V(tx, ty) is used for each pixel in the peripheral pixel region of FIG. 13, and returns "1" for pixels that are not saturated (including boundary pixels), and returns "0" for saturated pixels. Thereby, in the approximate formula (7-2), saturated pixels can be masked and the predicted GR ratio can be predicted using only non-saturated pixels.

[0104] The approximate formula (7-2) for calculating the predicted GR ratio indicates that an operation is performed on all peripheral pixels in the peripheral pixel region of FIG. 13 to calculate the sum. Term B of the approximate formula (7-2) represents the weight of the distance from the pixel of interest to the peripheral pixel of the saturation boundary, and a function in which the weight decreases according to the distance is used. In this embodiment, the same formula as the formula for the distance weight used in a bilateral filter or the like is used, but other mathematical formulas may be used as long as they are arithmetic expressions in which the weight decreases according to the distance.

[0105] Term D of the approximate formula (7-2) is a term for converging the approximate formula (7-2) to a preset fixed pixel value when the saturation boundary is not included in the peripheral pixel region. When all the pixels in the peripheral pixel region are saturated pixels and there is no saturation boundary, term C becomes "0". At this time, the calculation results of terms A, B, and C of the approximate formula (7-2) become "0", and term E also becomes "0". Thereby, the predicted value of the pixel of interest predicted by the approximate formula (7-2) becomes the fixed value "C". The fixed value C is preferably set to the white balance value WB described in FIG. 9. By using the parameters of the approximate formula (7-2) for the R component, the predicted GB ratio can be calculated.

[0106] Note that the coefficient a may be directly calculated by the least squares method. In the first term of Equation (2), since the coefficient a is the variable to be obtained, the coefficient a that minimizes the difference in the predicted pixel value G^ may be obtained from the pixels in the peripheral pixel region. Specifically, it is only necessary to calculate the coefficient a that minimizes the cost function J shown in Equation (7-3). Equation (7-3) is a calculation using the general least squares method, and can be calculated at high speed by using the method used in the guided filter.

[0107] The term A in Equation (7-3) represents the prediction error of the G value in unsaturated pixels and is the expression to be minimized. However, as described above, for example, when all the pixels in the peripheral pixel region are saturated, the function V(tx,ty) in Equation (7-1) becomes all "0", and no solution can be obtained.

[0108] To avoid this problem, terms B and C in Equation (7-3) are provided. Terms B and C in Equation (7-3) are regularization terms in the least squares method. Since the cost function J increases as the coefficient a deviates from the white balance value WB, the coefficient a is optimized to a value close to the white balance value WB. By this action, when the function V(tx,ty) is all "0", the coefficient a can be converged to the white balance value WB. The sign k of terms B and C in Equation (7-3) indicates the strength of the constraint condition.

[0109] Note that according to Equation (7-4), an appropriate value of the blending ratio α between the first prediction process (prediction by different color components) shown in FIG. 14 and the second prediction process (prediction by non-saturated self-color) shown in FIG. 15 can be dynamically calculated. For example, by optimizing an expression in a form that combines the first term and the second term of Equation (2) in FIG. 8, and calculating the coefficients a and b shown in Equation (7-4) using Equation (7-3), it is possible to eliminate the need to determine the blending ratio α. Here, F(x,y,G) in Equation (7-4) is calculated in advance using the approximate expression (8-3) in FIG. 15 described later.

[0110] FIG. 15 shows an example of an approximate expression when the prediction of the pixel value shown in FIG. 13 is performed using the second prediction process (prediction by non-saturated self-color). FIG. 15 shows an example of calculating the predicted pixel value of pixel G, but the predicted pixel values of pixel R and pixel B can be calculated in the same way. In FIG. 15, the pixel value of the target pixel is predicted using the boundary pixels among the pixels in the peripheral pixel region.

[0111] Generally, since there are a plurality of boundary pixels in the peripheral pixel region, each predicted pixel value is obtained by weighted averaging according to the distance from the saturated pixel. Also, when there is no boundary of the interval in the peripheral pixel region, Equation (8-2) is used to set the predicted pixel value to a fixed value.

[0112] Equation (8-1) is an equation for approximately calculating the derivative and is calculated for use in linear prediction. The function W(tx,ty) shown in Equation (8-2) returns "1" for non-saturated pixels (including boundary pixels) and "0" for saturated pixels, similar to Equation (7-1). By substituting the function V(tx,ty) into term C of the approximate equation (8-3), pixels other than boundary pixels can be masked and the pixel value can be predicted only from boundary pixels.

[0113] The approximate equation (8-3) for calculating the predicted value F(x, y, G) of G indicates that an operation is performed on all pixels in the peripheral pixel region of FIG. 13 and a sum is calculated. Term A of the approximate equation (8-3) represents the linear prediction value of the target pixel with respect to the boundary pixel. Term A of the approximate equation (8-3) is an equation obtained by expanding the one-dimensional linear prediction equation shown in FIG. 10 to two dimensions. Note that, similar to the first embodiment, an approximation may be made using a multi-dimensional equation such as a quadratic equation instead of a linear equation.

[0114] Term B of the approximate equation (8-3) represents the weight of the distance from the target pixel to the boundary pixel, and a function whose weight decreases according to the distance is used. In this embodiment, an equation similar to the equation of the distance weight used in a bilateral filter or the like may be used. Note that other mathematical expressions may be used as long as they are arithmetic expressions in which the weight decreases according to the distance.

[0115] Term D of the approximate equation (8-3) is a term for converging the approximate equation (8-3) to a fixed value when there are no boundary pixels in the peripheral pixel region, similar to term D of the approximate equation (7-2). The fixed value C is preferably set to the white balance value WB. Term E (denominator) of the approximate equation (8-3) is a normalization term for weighted averaging processing.

[0116] As described above, also in this embodiment, the same effects as those of the above-described embodiments can be obtained. Further, in this embodiment, the image processing apparatus 10 predicts a pixel value using boundary pixels extracted from the pixel values of the pixels in the peripheral pixel region around the target pixel for which the pixel value is predicted, and calculates a predicted pixel value by weighted-averaging the predicted pixel values according to the distance. Thereby, it is possible to eliminate the processing depending on the scanning direction and suppress the occurrence of artifacts. Further, when the peripheral pixel region does not include boundary pixels, by setting the predicted pixel value to a preset fixed pixel value, it is possible to correctly predict the pixel value in the saturation region even when boundary pixels cannot be extracted.

[0117] Note that, for example, when the image processing apparatus 10 predicts a pixel value for the current target image, it acquires an image and then calculates an approximation formula or the like. Therefore, it takes time from the acquisition of the image until the completion of the prediction of the pixel value, or until the display of the image on a display device or the like using the predicted pixel value from the acquisition of the image. Further, for example, when the imaging device 19 acquires an image of a subject by raster scanning as shown in FIG. 5, there may be a case where the parameters of the approximation formula used for calculating the predicted pixel value cannot be calculated until the image of the entire angle of view is acquired.

[0118] Therefore, the image processing apparatus 10 according to the first to third embodiments described above may calculate the parameters of the approximation formula using an image acquired before the immediately acquired image or the image currently being acquired. Then, the image processing apparatus 10 performs prediction of the pixel value in the currently acquired image using the calculated parameters. Thereby, it is possible to quickly perform the prediction process of the pixel value for the currently acquired image and improve the real-time performance. In particular, by applying it to fields where real-time performance of image processing is required, such as the imaging device 19 mounted on the moving body 200, it is possible to improve the recognition performance of the image and the like.

[0119] Although the present invention has been described based on the above embodiments, the present invention is not limited to the requirements shown in the above embodiments. Regarding these points, changes can be made without departing from the gist of the present invention, and can be appropriately determined according to the application form.

Explanation of Signs

[0120] 10 Image processing apparatus 10a Acquisition unit 10b Extraction unit 10c Prediction unit 10d Output unit 11 Information processing apparatus 12 Display apparatus 14 Wireless communication apparatus 15 Sensor 16 Driving apparatus 17 Lamp apparatus 18 Navigation apparatus 19 (19A, 19B, 19C, 19D, 19E) Imaging apparatus 100 Image processing system 200 Moving body BUS Bus S Sun

Claims

1. An extraction unit that extracts, from image data indicating an image including a plurality of pixels acquired by an imaging device, an unsaturated region where pixel values are not saturated and a saturated region where pixel values are saturated; A prediction unit that predicts a pixel value of a target pixel in the saturated region based on a pixel value of a boundary pixel in a boundary region between the saturated region and the unsaturated region in the unsaturated region; and has the image data is acquired for each pixel of a plurality of colors, the extraction unit extracts the unsaturated region and the saturated region from each of the image data of the plurality of colors, the prediction unit executes a first prediction process for predicting a pixel value of the saturated region of the target color based on a ratio between a pixel value of the target color in the boundary region of the target color, which is any one of the plurality of colors, and a pixel value of another color that is not saturated; changing the ratio P1 / P2 between the pixel value P1 of the target color and the pixel value P2 of another color that is not saturated so as to converge to a white balance value of the other color at a pixel position in an all-saturated region where all pixel values of the plurality of colors are saturated; An image processing apparatus.

2. The prediction unit executes a second prediction process for predicting a pixel value of the target pixel based on a change amount of pixel values of a plurality of the boundary pixels of the same color as the pixel for predicting the pixel value along a direction toward the saturated region. The image processing apparatus according to claim 1.

3. The prediction unit executes a second prediction process for predicting a pixel value of the target pixel based on a change amount of pixel values of a plurality of the boundary pixels of the same color as the pixel for predicting the pixel value along a direction toward the saturated region, and mixes the pixel value predicted by the first prediction process and the pixel value predicted by the second prediction process at a predetermined ratio. The image processing apparatus according to claim 1.

4. The prediction unit predicts a pixel value of the target pixel by weighted-averaging a plurality of pixel values of the target pixel respectively predicted based on pixel values of a plurality of the boundary pixels according to distances from the plurality of the boundary pixels to the target pixel. The image processing apparatus according to any one of claims 1 to 3.

5. The extraction unit extracts the boundary region from a peripheral pixel region including the target pixel, and the prediction unit predicts a plurality of pixel values of the target pixel respectively based on pixel values of a plurality of the boundary pixels in the boundary region of the peripheral pixel region. The image processing apparatus according to claim 4.

6. When the boundary region is not included in the peripheral pixel region, the prediction unit uses a preset pixel value as a predicted value of the target pixel. The image processing apparatus according to claim 5.

7. After predicting the pixel values of the saturation region in the image data, the pixel values of each pixel of the image data are compressed so that the maximum value of the predicted pixel values falls within the pixel values of the non-saturation region. The image processing apparatus according to any one of claims 1 to 6.

8. The prediction unit performs a process of calculating parameters of an approximation formula used for predicting the pixel value of the pixel of interest based on the extraction result of the extraction unit using the image data previously acquired by the imaging device, and performs a process of predicting in real time the pixel value of the pixel of interest in the image data acquired by the imaging device using the calculated parameters. The image processing apparatus according to any one of claims 1 to 7.

9. An extraction process of extracting a non-saturation region where pixel values are not saturated and a saturation region where pixel values are saturated from image data showing an image including a plurality of pixels acquired by an imaging device, and a prediction process of predicting the pixel values of the saturation region based on the pixel values of boundary pixels of a boundary region between the saturation region and the non-saturation region in the non-saturation region, which is an image processing method for performing wherein the image data is acquired for each pixel of a plurality of colors, the extraction process extracts the non-saturation region and the saturation region from each of the image data of the plurality of colors, the prediction process executes a first prediction process of predicting the pixel values of the saturation region of the target color based on the ratio between the pixel value of the target color in the boundary region of the target color, which is any one of the plurality of colors, and the pixel values of other colors that are not saturated, and changes the ratio P1 / P2 between the pixel value P1 of the target color and the pixel value P2 of other colors that are not saturated so as to converge to the white balance value of the other color at any pixel position in the fully saturated region where all pixel values of the plurality of colors are saturated. Image processing method.

Images