Image processing apparatus and image processing method

Abstract

An image processing apparatus includes an extraction section and a prediction section. The extraction section extracts a non-saturated region in which pixel values are not saturated and a saturated region in which pixel values are saturated from image data representing an image including a plurality of pixels acquired by an image pickup apparatus. The prediction section predicts a pixel value of a pixel of interest of the saturated region based on a pixel value of a boundary pixel of a boundary region of the saturated region and the non-saturated region. Thus, it is possible to predict a luminance value of an object image of a dead white occurrence region using image data in which dead white has occurred.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and an image processing method. Background Technology

[0002] Recently, various methods have been proposed to improve the recognizability of images displayed on display devices by performing image processing on image data acquired by camera devices mounted on moving bodies such as vehicles. For example, by synthesizing multiple image data acquired by multiple camera devices, or by selecting any one of multiple image data, it is possible to display an image that suppresses white or black areas.

[0003] [Existing Technical Documents]

[0004] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-162034

[0006] [Patent Document 2] Japanese Patent Application Publication No. 2020-166628

[0007] [Patent Document 3] Japanese Patent Application Publication No. 2018-032952

[0008] [Patent Document 4] Japanese Patent Application Publication No. 2017-112421

[0009] [Patent Document 5] Japanese Patent Application Publication No. 2018-125585

[0010] [Patent Document 6] Japanese Patent Application Publication No. 2012-134586

[0011] [Patent Document 7] Japanese Patent Application Publication No. 2013-093786 Summary of the Invention

[0012] [The problem the invention aims to solve]

[0013] For example, the pixel values ​​of image data in areas where blown-out areas occur become the upper limit that the camera device can output. However, the actual brightness of the photographic area of ​​the subject with blown-out areas is not uniform but varies gradually. Therefore, a method that uniformly corrects saturation areas with pixel values ​​exceeding the upper limit cannot obtain an accurate image of the subject. Furthermore, it is difficult to predict the image of the subject in areas with blown-out areas from image data acquired by a single camera device.

[0014] The present invention was made in view of the above-mentioned problems, and its object is to predict the brightness value of the subject in the area where the blown area occurs using image data in which blown areas occur.

[0015] [Methods for solving the problem]

[0016] According to one aspect of the present invention, an image processing apparatus includes: an extraction unit that extracts unsaturated regions with unsaturated pixel values ​​and saturated regions with saturated pixel values ​​from image data representing an image containing a plurality of pixels acquired by a camera device; and a prediction unit that predicts the pixel value of a pixel of interest in the saturated region based on the pixel value of a boundary pixel in the unsaturated region that is a boundary region between the unsaturated region and the saturated region.

[0017] [The effects of the invention]

[0018] According to publicly available technology, it is possible to predict the brightness value of the subject image in the area where the blown-out occurred using image data that has been blown-out. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating an example of an image processing system that includes the image processing apparatus of the first embodiment.

[0020] Figure 2 It means Figure 1 A block diagram illustrating an example of the functional structure of an image processing device.

[0021] Figure 3 It means that it is mounted on Figure 1 A schematic block diagram outlining the structure of various devices on a moving body.

[0022] Figure 4 It means Figure 3 A block diagram illustrating the structure of an image processing device and an information processing device.

[0023] Figure 5 It means Figure 1 An illustration of an example of image processing of images acquired by a camera device.

[0024] Figure 6 It means that by Figure 3 An illustrative diagram showing examples of pixel values ​​that the camera device can output and pixel values ​​that the camera device cannot output from the image data of the subject acquired by the camera device.

[0025] Figure 7 It means Figure 3 A flowchart illustrating an example of an image value prediction method implemented by an image processing device.

[0026] Figure 8 It means in Figure 7 An illustration of an example of an approximation used in image processing.

[0027] Figure 9 It means through Figure 8 An illustrative diagram illustrating an example of the first prediction process predicting pixel values.

[0028] Figure 10 It means through Figure 8 An illustrative diagram illustrating an example of the second prediction process predicting pixel values.

[0029] Figure 11 This is an illustrative diagram illustrating an example of a method for compressing pixel values ​​predicted using at least one of the first prediction process and the second prediction process.

[0030] Figure 12 This is an explanatory diagram illustrating an example of image processing performed by the image processing apparatus in the second embodiment.

[0031] Figure 13 This diagram illustrates an example of image processing performed by the image processing apparatus in the third embodiment.

[0032] Figure 14 This indicates that the first prediction process is implemented. Figure 13 The illustration shows an example of an approximation for the prediction of pixel values.

[0033] Figure 15 This indicates that the second prediction process is implemented. Figure 13 The illustration shows an example of an approximation for the prediction of pixel values. Detailed Implementation

[0034] The following description uses the accompanying drawings to illustrate the embodiments. In the following description, image data will sometimes be simply referred to as an image.

[0035] (First Embodiment)

[0036] Figure 1 This describes an example of an image processing system that includes the image processing apparatus of the first embodiment. Figure 1 The image processing system 100 shown is mounted, for example, on a mobile vehicle 200 such as a car. Cameras or other imaging devices 19A, 19B, 19C, 19D, and 19E are provided in front of, behind, to the left and right of, and inside the vehicle 200 in the direction of travel D of the mobile vehicle 200. Hereinafter, without distinguishing between the imaging devices 19A, 19B, 19C, 19D, and 19E, they will also be referred to as imaging device 19.

[0037] Furthermore, the number and location of the camera devices 19 installed on the moving body 200 are not limited to... Figure 1For example, one camera device 19 may be installed at the front of the mobile body 200, or two camera devices 19 may be installed at the front and rear. Alternatively, the camera device 19 may be installed on the roof of the mobile body 200. Furthermore, the mobile body 200 equipped with the image processing system 100 is not limited to automobiles; it may also be a handling robot or drone operating in a factory. In addition, the image processing system 100 may be a system that processes images acquired from camera devices other than those installed on the mobile body 200, such as surveillance cameras, digital still cameras, or video recorders.

[0038] The image processing system 100 includes an image processing device 10, an information processing device 11, and a display device 12. Additionally, in Figure 1 In this illustration, for ease of understanding, the image processing system 100 is superimposed on a schematic diagram of the mobile body 200 viewed from above. However, in reality, the image processing device 10 and the information processing device 11 are mounted on a control board or similar component mounted on the mobile body 200, and the display device 12 is positioned in a location visible to a person within the mobile body 200. Alternatively, the image processing device 10 may also be mounted on the control board or similar component as part of the information processing device 11. The image processing device 10 is connected to each camera device 19 via a signal line or wirelessly.

[0039] Figure 2 express Figure 1 An example of the functional structure of the image processing apparatus 10. The image processing apparatus 10 includes an acquisition unit 10a, an extraction unit 10b, a prediction unit 10c, and an output unit 10d. The acquisition unit 10a acquires image data representing images of the area around the moving body 200 captured by each imaging device 19. The extraction unit 10b and the prediction unit 10c perform image processing (correction processing) on ​​the image data acquired from each imaging device 19. The output unit 10d outputs the result of the image processing to at least one of the display device 12 and the information processing device 11.

[0040] Return to Figure 1 The display device 12 may be, for example, a side mirror monitor, a rearview mirror monitor, or a display of a navigation device mounted on the moving body 200. Alternatively, the display device 12 may be a display mounted on the dashboard, or a head-up display that projects images onto a projection panel or windshield.

[0041] The information processing device 11, including a computer such as a processor, performs recognition processing based on image data received via the image processing device 10. For example, the information processing device 11 mounted on the mobile body 200 performs image data recognition processing to detect other mobile bodies, signals, signs, white lines on the road, and people, and determines the situation around the mobile body 200 based on the detection results. In addition, the information processing device 11 may also include an automatic driving control device that controls the movement, stopping, right turn, and left turn of the mobile body 200.

[0042] Figure 3 Indicates that it is mounted on Figure 1 A summary of the structure of various devices on the mobile body 200. The mobile body 200 includes an image processing device 10, an information processing device 11, a display device 12, at least one electronic control unit (ECU) 13, and a wireless communication device 14 interconnected via an internal network. The mobile body 200 also includes sensors 15, a drive device 16, a lighting device 17, a navigation device 18, and a camera device 19. For example, the internal network is a CAN (Controller Area Network) or Ethernet (registered trademark) vehicle network.

[0043] The image processing apparatus 10 corrects the image data (frame data) acquired by the imaging device 19 to generate corrected image data. The image processing apparatus 10 may also record the generated corrected image data in an external or internal recording device.

[0044] The information processing device 11 can function as a computer controlling various parts of the mobile body 200. The information processing device 11 controls the entire mobile body 200 by controlling the ECU 13. The information processing device 11 can identify objects outside the mobile body 200 based on images generated by the image processing device 10, and can track the identified objects.

[0045] Display device 12 displays images generated by image processing device 10, as well as corrected images. Display device 12 can also display an image of the moving body 200 in the backward direction in real time when the moving body 200 is moving backward. Furthermore, display device 12 can display images output from navigation device 18.

[0046] Each ECU 13 is correspondingly provided with a mechanism such as an engine or transmission. Each ECU 13 controls its corresponding mechanism based on instructions from the information processing device 11. The wireless communication device 14 communicates with external devices of the mobile body 200. The sensor 15 is a sensor that detects various types of information. For example, the sensor 15 may include a position sensor that acquires the current position information of the mobile body 200. In addition, the sensor 15 may also include a speed sensor for detecting the speed of the mobile body 200.

[0047] The drive unit 16 is a variety of devices used to move the moving body 200. The drive unit 16 may include, for example, an engine, a steering device (steering gear), a braking device (brake), etc. The lighting device 17 is a variety of lamps mounted on the moving body 200. The lighting device 17 may include, for example, headlights, turn signals, taillights, and brake lights. The navigation device 18 is a device that guides the user to a destination via sound and display.

[0048] Figure 4 express Figure 3 An example of the structure of the image processing apparatus 10 and the information processing apparatus 11. Since the structures of the image processing apparatus 10 and the information processing apparatus 11 are the same, the structure of the image processing apparatus 10 will be described below. For example, the image processing apparatus 10 includes a CPU 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a storage device 24 that are interconnected via a bus.

[0049] CPU 20 performs various image processing tasks, described later, by executing image processing programs stored in storage device 24. Interface device 21 is used to connect to a network (not shown). Auxiliary storage device 23, such as HDD (Hard Disk Drive) or SSD (Solid State Drive), holds image processing programs, image data, and various parameters used in image processing.

[0050] The storage device 24, such as DRAM (Dynamic Random Access Memory), stores image processing programs transferred from the auxiliary storage device 23. The drive device 22 has an interface for connecting to the recording medium 30, and for example, forwards the image processing programs stored in the recording medium 30 to the auxiliary storage device 23 according to instructions from the CPU 20. Additionally, the drive device 22 can also transfer image data stored in the auxiliary storage device 23 to the recording medium 30.

[0051] Figure 5 express Figure 1 This is an example of image processing of an image IMG acquired by the camera device 19. For example, the camera device 19 is an image sensor in which pixels formed by three photoelectric conversion elements that detect light of red (R), green (G), and blue (B) are arranged in a matrix. In the following text, each of the three photoelectric conversion elements is also referred to as a pixel. The individual pixel values ​​of the red (R), green (G), and blue (B) pixels are also referred to as R value, B value, and G value, respectively.

[0052] Furthermore, the photoelectric conversion elements included in the pixels of the imaging device 19 are not limited to red (R), green (G), and blue (B). For example, the pixels of the imaging device 19 may include photoelectric conversion elements for detecting near-infrared light. Alternatively, the pixels of the imaging device 19 may include photoelectric conversion elements of cyan, yellow, and magenta, or they may include photoelectric conversion elements of cyan, yellow, and red.

[0053] The dotted-line arrows in the image represent the image data of the subject acquired by the imaging device 19 through raster scanning. Furthermore, the image IMG can be acquired by any of the other imaging devices 19A-19D, or it can be a composite image created by combining images acquired by multiple imaging devices 19.

[0054] Figure 5 The image IMG is a high-brightness subject image, such as the sun S. Therefore, the brightness increases in the pixels closer to the sun S. Figure 5 The image IMG is divided into 4 intervals, with the curve shown as a dashed line as the boundary.

[0055] In the first interval, which is farthest from the sun S, the R, G, and B values ​​are below the maximum pixel value (upper limit) that the camera device 19 can obtain (no saturation). That is, the first interval is a non-saturated region where the R, G, and B values ​​are not saturated.

[0056] In the second interval, which is the second farthest from the sun (S), the R and B values ​​are below the upper limit, while the G value is above the upper limit (G saturation). That is, the second interval is an unsaturated region where the R and B values ​​are not saturated, and a saturated region where the G value is saturated.

[0057] In the third interval, which is closer to the Sun's S value than the second interval, the B value is below the upper limit, while the G and R values ​​are above the upper limit (GR saturation). That is, the third interval is an unsaturated region that is not saturated at the B value, and a saturated region that is saturated at the G and R values ​​respectively.

[0058] In the fourth interval, closest to the sun S, the G, R, and B values ​​exceed the upper limit (GRB saturation). That is, the fourth interval is a fully saturated region where all G, R, and B values ​​are saturated. When pixel values ​​are saturated, the image acquired by the imaging device 19 cannot accurately represent the colors of the subject. The image, with all G, R, and B values ​​saturated, becomes white, producing what is known as a "dead white" image.

[0059] The image processing apparatus 10 performs pixel value extraction processing using image data acquired for each pixel of multiple colors from the imaging device 19, extracting unsaturated and saturated regions for each R, G, and B value. The image processing apparatus 10 performs at least one of a first prediction processing that predicts pixel values ​​using the ratio of unsaturated base color components to different color components, and a second prediction processing that predicts pixel values ​​using unsaturated base color components.

[0060] In the first prediction process, the image processing device 10 predicts the G value of the second interval based on the R value of the second interval, for example, based on the ratio of the G value to the R value in the first interval where the G value and R value are not saturated. Additionally, the image processing device 10 predicts the G value of the second interval from the B value of the third interval, for example, based on the ratio of the G value to the B value in the first interval where the G value and B value are not saturated.

[0061] Image processing apparatus 10, for example, predicts the R value of the fourth interval based on the predicted R value of the fourth interval and the ratio of the R value to the B value in the interval where the B value is not saturated. Additionally, image processing apparatus 10, for example, predicts the G value of the fourth interval based on the predicted B value of the fourth interval and the ratio of the G value to the B value in the first interval where the B value is not saturated.

[0062] In the second prediction process, the image processing device 10 predicts, for example, the pixel values ​​in the saturated region of G value (from the second to the fourth interval) based on the pixel values ​​of multiple boundary pixels of G color located in the boundary region between the first interval (where G value is not saturated) and the second interval (where R value is saturated). Additionally, the image processing device 10 predicts, for example, the pixel values ​​in the saturated region of R value (from the third to the fourth interval) based on the pixel values ​​of multiple boundary pixels of R color located in the boundary region between the second interval (where R value is not saturated) and the third interval (where R value is saturated).

[0063] Furthermore, the image processing apparatus 10 predicts the pixel values ​​in the saturated region (fourth interval) of B value based, for example, on the pixel values ​​of multiple boundary pixels of B color located in the boundary region of the fourth interval (saturated region) in the third interval where B value is not saturated. Using Figures 8 to 10 Explain specific examples of the first and second prediction processes.

[0064] By performing at least one of the first prediction processing and the second prediction processing, the image processing apparatus 10 is able to calculate a pixel value corresponding to the actual brightness value of the subject as a predicted value, even when receiving pixel values ​​in a saturated state from the imaging device 19. Thus, the image processing apparatus 10 can correct image data containing pixel values ​​of areas with blown-out areas to image data without blown-out areas.

[0065] In addition, Figure 5 and Figure 6 The following examples illustrate how pixel values ​​saturate in the order of G, R, and B values. This is because, generally speaking, in RGB image sensors, sensitivity decreases from high to low in the order of G, R, and B components. However, the order in which pixel values ​​saturate colors is not limited to this. Figure 5 The example shown.

[0066] Figure 6 Indicated by Figure 3 The image data of the subject acquired by the camera device 19 includes the pixel values ​​that the camera device 19 can output and the pixel values ​​that the camera device 19 cannot output. Figure 6 The first to fourth intervals shown correspond to respectively Figure 5 The first to the fourth interval. For example, Figure 6 express Figure 5 The G, R, and B values ​​in the raster scan (5). Figure 6 The horizontal axis represents the position of the pixel on the raster scan (5) (horizontal coordinate).

[0067] Figure 6 The vertical axis is represented by a logarithmic axis, showing the ratio of pixel values ​​when the maximum pixel value achievable by the imaging device 19 is set to "1". For example, when the imaging device 19 outputs pixel values ​​in 8 bits, "1" on the vertical axis corresponds to a pixel value of "255". When the brightness of the subject exceeds the maximum pixel value (vertical axis "1"), the imaging device 19 outputs that maximum pixel value (vertical axis "1"). In the following text, the maximum pixel value is also referred to as the upper limit value.

[0068] exist Figure 6 In the image processing diagram, the solid lines represent the G, R, and B values, which are the pixel values ​​output by the imaging device 19. The dashed lines represent the G, R, and B values, which are the pixel values ​​corresponding to the actual brightness of the subject. The image processing device 10 predicts pixel values ​​exceeding the upper limit values ​​represented by the dashed lines by performing at least one of the first prediction processing and the second prediction processing.

[0069] Figure 7 express Figure 3 An example of an image value prediction method implemented by the image processing apparatus 10. That is, Figure 7 This illustrates an example of an image processing method based on the image processing apparatus 10. For instance, it is implemented by executing an image processing program via the CPU 20 of the image processing apparatus 10. Figure 7 The process is shown. A specific example of the image processing method implemented by the image processing device 10 will be described later. Figure 8 This will be explained later.

[0070] in addition, Figure 7The process shown can also be implemented using hardware such as an FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) mounted in the image processing device 10. Alternatively, Figure 7 The process shown can be achieved through the collaboration of hardware and software.

[0071] For example, for Figure 5 Each pixel of the raster scan row shown is executed Figure 7 The process shown is one loop. Additionally, in Figure 5 In the image IMG shown, with the sun S located in the left corner of the image IMG, sections 1 to 4 are arranged from right to left. Additionally, in... Figure 5 In the image IMG shown, when the sun S is located near the center of the image IMG, there are intervals 1 to 4 arranged from left to right and intervals 1 to 4 arranged from right to left in the image IMG.

[0072] First, in step S10, if a boundary between the first and second intervals is detected based on at least one of the G, R, and B values, the image processing apparatus 10 executes steps S12 and S20. If no boundary between the first and second intervals is detected, the image processing apparatus 10 executes step S40.

[0073] In step S12, the image processing apparatus 10 generates an approximation formula by calculating parameters for an approximation formula used to calculate pixel values ​​exceeding the upper limit value in the second interval. Next, in step S14, the image processing apparatus 10 performs image processing to calculate pixel values ​​exceeding the upper limit value using the approximation formula with the calculated parameters set. Afterward, the processing proceeds to step S40.

[0074] In step S20, if the boundary between the second and third intervals is detected based on at least one of the G, R, and B values, the image processing apparatus 10 executes steps S22 and S30. If the boundary between the second and third intervals is not detected, the image processing apparatus 10 executes step S40.

[0075] In step S22, the image processing apparatus 10 generates an approximation formula by calculating parameters for an approximation formula used to calculate pixel values ​​exceeding the upper limit value in the third interval. Next, in step S24, the image processing apparatus 10 performs image processing to calculate pixel values ​​exceeding the upper limit value using the approximation formula with the calculated parameters set. Afterward, the processing proceeds to step S40.

[0076] In step S30, when the boundary between the third and fourth intervals is detected based on at least one of the G, R, and B values, the image processing apparatus 10 executes step S32. When no boundary between the third and fourth intervals is detected, the image processing apparatus 10 executes step S40.

[0077] In step S32, the image processing apparatus 10 generates an approximation formula by calculating parameters for an approximation formula used to calculate pixel values ​​exceeding the upper limit value in the fourth interval. Next, in step S34, the image processing apparatus 10 performs image processing to calculate pixel values ​​exceeding the upper limit value using the approximation formula with the calculated parameters set. Afterward, the processing proceeds to step S40.

[0078] In step S40, the image processing device 10 returns to step S10 if image processing continues, and ends if image processing does not continue, for example, when processing of one frame is completed. Figure 7 The processing is shown. In addition, the image processing apparatus 10 may also perform image processing such as noise removal processing or edge enhancement processing after step S40 and before returning to step S10.

[0079] Figure 8 Indicates in Figure 7 This is an example of an approximation used in image processing. In the formulas for each interval, the symbol ^ appended to the symbols G, R, and B indicates the predicted pixel value. In the following text, the predicted pixel value is represented by the pixel values ​​G^, R^, and B^. Dashed underlines indicate examples of approximations used in the first prediction process based on predictions of different color components. Dotted underlines indicate examples of approximations used in the second prediction process based on predictions of unsaturated base color components. Figure 9 This represents an example of predicting pixel values ​​using the first prediction process. Figure 10 This represents an example of predicting pixel values ​​through the second prediction process.

[0080] The symbol α represents the applicable ratio for the first prediction process, and the symbol 1-α represents the applicable ratio for the second prediction process. That is, the symbol α represents the mixing ratio at which the pixel values ​​predicted in the first prediction process and the pixel values ​​predicted in the second prediction process are mixed at a specified ratio. When α = 1, only the first prediction process (prediction based on different color components) is applied, and when α = 0, only the second prediction process (prediction based on unsaturated base color components) is applied.

[0081] By applying a mixing rate α, the predicted pixel values ​​based on the first prediction process and the predicted pixel values ​​based on the second prediction process can be appropriately mixed according to the features of the subject (image data), thereby improving the prediction accuracy of pixel values ​​in unsaturated regions.

[0082] In each formula, R represents the pixel value of the R component, and B represents the pixel value of the B component. Coefficient a R This represents the coefficient used to calculate the pixel value R^ based on the pixel value R or the pixel value R^. Coefficient a B This represents the coefficient used to calculate pixel value G^ or pixel value R^ based on pixel value B or pixel value B^. For example, coefficient a R a B And the mixing ratio α is a parameter set for each approximation formula to calculate pixel values ​​G^, R^, B^.

[0083] As shown in equation (5-1), the coefficient a R This represents the predicted GR ratio. The predicted GR ratio is the ratio of the G value (P1) of the color of interest G in the first interval to the R values ​​(P2) of other colors R, G value / R value (P1 / P2). As shown in formula (5-2), the coefficient a B This represents the predicted GB ratio. The predicted GB ratio is the ratio of the G value (P1) of color G to the B value (P2) of other colors B, where G value / B value (P1 / P2) is the ratio. The predicted GR ratio and predicted GB ratio are preferably calculated at the boundary between the pixel values ​​with the largest difference in the first interval and the second interval.

[0084] The symbol β represents the application ratio of formulas (5-1) and (5-2). That is, the symbol β represents the priority (weight) of the predicted GR ratio and the predicted GB ratio. When β = 1, only the predicted GR ratio is applied, and when β = 0, only the predicted GB ratio is applied. In addition, the difference between pixel values ​​G and B is greater than the difference between pixel values ​​G and R, so using the predicted GB ratio makes it easier to improve the prediction accuracy of pixel value G^. Therefore, it is preferable to decrease β so that the weight of the predicted GB ratio is greater than the weight of the predicted GR ratio.

[0085] In this way, by applying the priority rate β, we can appropriately set which of the pixel values ​​R and B to use in combination with the characteristics of the subject (image data), which can improve the prediction accuracy of pixel values ​​in unsaturated areas.

[0086] coefficient a C Since it is not used in this embodiment, it is set to "0" as shown in formula (5-3). In addition, the predicted RB ratio of R value / B value, which is the ratio of R value to B value in the first interval, can also be used in the approximation formula.

[0087] In the fourth interval where pixel values ​​of all colors are saturated, the first prediction process cannot be applied before predicting a pixel value. Therefore, in the fourth interval, as shown in equation (4-1), the image processing device 10 initially uses the second prediction process to calculate the pixel value B^, and uses the calculated pixel value B^ to calculate the pixel values ​​R^ and G^.

[0088] Equation (4-1) replaces the symbol G in equation (6-2) with the symbol B. Equations (6-1) and (6-2) represent examples of using the components of G to predict the G value in the saturation region. In equations (4-1), (6-1), (6-2), and (6-3), the symbol x represents the horizontal direction of the image ( Figure 5 The coordinates of the image (raster scanning direction), with the symbol y representing the coordinates of the vertical direction of the image.

[0089] Equation (6-1) represents the pixel value G(t) of the boundary pixel adjacent to the saturation range of the pixel value. x The pixel values ​​G(t) of other boundary pixels, y) x The difference between (-1, y) is used as the derivative of the pixel value. Boundary pixels are included in the non-saturated region (e.g., Figure 6 In the first interval). Symbol t x The symbol t represents the horizontal position (coordinates) of the boundary pixel. x -1 indicates the horizontal position (coordinates) of the pixel immediately preceding the pixel in the boundary pixel that is adjacent to the saturated region.

[0090] The first term of equation (6-2) represents the pixel value G(t) of the boundary pixel adjacent to the saturation interval. x The second term of equation (6-2) represents the predicted pixel value x and the boundary pixel t. x The product of the horizontal distance and the differential value obtained from formula (6-1). That is, the second term of formula (6-2) represents the distance from the boundary pixel t. x The amount of change in pixel value to the predicted pixel x (predicted value).

[0091] Equations (6-1) and (6-2) represent examples of linear prediction of pixel values ​​using first-order differentiation. However, as shown in equation (6-3), pixel values ​​can also be approximated using a quadratic function by using second-order differentiation. Furthermore, this can be extended to an nth-order function. In this case, it becomes related to the boundary pixel t. x The changes in pixel values ​​are processed using Taylor expansion equivalents.

[0092] Figure 9 Indicates passage Figure 8 The first prediction process described herein is an example of predicting pixel values. In the first prediction process, the image processing device 10 uses different color components to predict the pixel value of the base color. Figure 9 middle, Figure 8 The coefficient α is set to "0". The image processing device 10 calculates the predicted GR ratio (G value / R value) and the predicted GB ratio (G value / B value) in the first interval where all color pixels are unsaturated, using G value, R value, and B value.

[0093] In the second interval, at each pixel position, the image processing device 10 uses Figure 8 The first term of equation (2) is obtained by using the coefficient a R The product of pixel value R and coefficient a B The pixel value G^ is calculated by adding the product of the pixel value B and the pixel value B.

[0094] In the third interval, the image processing device 10 first uses at each pixel position... Figure 8 The first term of the expression (3-1) is based on the coefficient a. B The pixel value R^ is calculated by multiplying the pixel value B and the pixel value B. Then, the image processing device 10 uses... Figure 8 The first term of the equation (3-2) is obtained by using the coefficient a R The product of the pixel value R^ and the coefficient a B The pixel value G^ is calculated by adding the product of the pixel value B and the pixel value B.

[0095] In the fourth interval, the image processing device 10 uses at each pixel position Figure 8 The pixel value B^ is calculated using formula (4-1). Next, the image processing device 10 uses... Figure 8 The first term of the expression (4-2) is based on the coefficient a. B The pixel value R^ is calculated by multiplying the pixel value B^ by the pixel value B^. Next, the image processing device 10 uses... Figure 8 The first term of the equation (4-3) is obtained by using the coefficient a R and the product of pixel value R^ and coefficient a B The pixel value G^ is calculated by adding the product of the pixel value B^ and the pixel value B^. In this way, through the first prediction process, the pixel value in the saturated region can be predicted using the ratio of different pixel values ​​in the unsaturated region.

[0096] Therefore, the parameters of the approximation formula are obtained in the first prediction process. Furthermore, the further the pixel values ​​G^, R^, and B^ predicted by the image processing device 10 are from the unsaturated region, the more likely the prediction accuracy is to decrease. Therefore, for example, the image processing device 10 corrects the predicted GR ratio and predicted GB ratio in a manner that the further away from the unsaturated region, the closer they are to non-color. Figure 9 The lower side represents the corrected curves for the predicted GR ratio and the predicted GB ratio.

[0097] The correction curve for the predicted GR ratio is set such that the predicted GR ratio becomes the white balance value (WB) of the R component at a specified pixel position in the 4th interval of RGB full color saturation. The correction curve for the predicted GB ratio is set such that the predicted GB ratio becomes the white balance value of the B component at a specified pixel position in the 4th interval. For example, the white balance calculation for the B value is the product of the B value and the white balance value of the B component; therefore, after white balance correction, the G value = the B value (non-color).

[0098] Therefore, even in the fourth interval where G, R, and B values ​​are all saturated, it is possible to suppress, for example, the color of the image after pixel value correction from deviating from the actual color of the subject. As a result, image data containing pixel values ​​G^, R^, and B^ predicted through the first prediction process, as described later... Figure 11 When compressed as described and displayed on the display device 12, an image without any sense of incongruity can be displayed.

[0099] Furthermore, the G, R, and B values ​​output from the camera device 19 are fixed at their upper limits (e.g., 255) after saturation. Therefore, the actual GR ratio gradually decreases in the second interval, reaching 1.0 at the end of the second interval. The actual GB ratio gradually decreases in both the second and third intervals, becoming 1.0 at the end of the third interval.

[0100] Figure 10 Indicates passage Figure 8 This is an example of the second prediction process for predicting pixel values. In the second prediction process, the image processing device 10 uses the base color component of the unsaturated region to predict the base color pixel value. Figure 10 The diagram shows the prediction method for pixel value G^, but the prediction methods for pixel values ​​R^ and B^ are the same as those for pixel value G^.

[0101] First, such as Figure 8 As shown in equation (6-1), the image processing apparatus 10 calculates the differential value dG / dx of the slope of the change in G value at the boundary (saturation boundary) between the first and second intervals where G value saturation occurs. Then, the image processing apparatus 10 applies the calculated differential value dG / dx to... Figure 8 Formula (6-2) is used to predict the pixel value G^ at any pixel position x in the second, third, and fourth intervals. Thus, through the second prediction process, the pixel value can be predicted using the rate of change of the pixel value of the intrinsic color component.

[0102] Figure 11 This illustrates an example of a method for compressing pixel values ​​predicted using at least one of the first and second prediction processes. If the predicted pixel value exceeds the upper limit of pixel values ​​that can be displayed on the display device 12 provided on the moving body 200, the image displayed on the display device 12 becomes a blank white. Therefore, the image processing apparatus 10 has the function of compressing the predicted pixel values. Figure 11 In this context, it is assumed that the maximum pixel value that can be displayed on the display device 12 is "255", and the maximum predicted pixel value is "512".

[0103] The image processing apparatus 10 uses a predetermined conversion formula corresponding to the maximum value of the predicted pixel value to convert the predicted pixel value into a pixel value that can be displayed on the display device 12. That is, after predicting the pixel values ​​of the saturated region in the image data, the image processing apparatus 10 compresses the pixel values ​​of each pixel in the image data so that the maximum value of the predicted pixel value converges to the pixel value of the unsaturated region.

[0104] Then, the image processing device 10 outputs the converted pixel values ​​as output pixel values ​​to the display device 12. Thus, even if the predicted pixel value exceeds the maximum pixel value that can be displayed on the display device 12, an image with suppressed white areas can be displayed on the display device 12.

[0105] Furthermore, the parameters in the transformation formula that transforms the predicted pixel value into the output pixel value can be changed according to the maximum value of the predicted pixel value. Additionally, to prevent hue changes in the image displayed on the display device 12, for example, if pixel value G^ is compressed, it is preferable that unsaturated pixel values ​​R and B are also compressed.

[0106] The image processing device 10 can also arbitrarily set the range for compressing pixel values ​​within an image frame. For example, in Figure 5 In this process, the image processing apparatus 10 may also compress only the pixel values ​​of empty regions in the image. Furthermore, the image processing apparatus 10 may divide the image into partial images of a predetermined size, and compress the pixel values ​​of the partial images containing pixels with saturated pixel values. Additionally, the image processing apparatus 10 may suppress unnatural-looking changes in pixel values ​​by smoothing the boundary between compressed and uncompressed regions.

[0107] In addition, in the case of Figure 3 When the information processing device 11 uses image data containing predicted pixel values ​​to perform image recognition processing, the image processing device 10 may also output the image data containing predicted pixel values ​​to the information processing device 11 without compression. In this case, the image processing device 10 may also output the image data containing predicted pixel values ​​via... Figure 4 The auxiliary storage device 23 or storage device 24 outputs to the information processing device 11.

[0108] In this embodiment, even if the pixel value of the subject acquired by the imaging device 19 exceeds the upper limit of the pixel value that the imaging device 19 can acquire, the correct pixel value can still be predicted. For example, the image processing device 10 can predict the pixel value in the saturated region by using the ratio of different pixel values ​​in the unsaturated region through the first prediction processing. In addition, the image processing device 10 can predict the pixel value in the saturated region by using the rate of change of the pixel value of the base color component through the second prediction processing.

[0109] Therefore, even when receiving saturated pixel values ​​from the imaging device 19, the image processing apparatus 10 can calculate a pixel value corresponding to the actual brightness value of the subject as a predicted value. As a result, the image processing apparatus 10 can correct image data containing pixel values ​​of areas with whiteouts to image data without whiteouts, and can perform high-precision image recognition processing using the corrected image data.

[0110] Furthermore, by compressing the pixel values ​​of image data, including the corrected pixel values, an image of the subject without white casts can be displayed on the display device 12, even when the subject is bright. As a result, the quality of the image displayed on the display device 12 can be improved.

[0111] By applying a mixing ratio α, the image processing apparatus 10 can appropriately mix the predicted pixel values ​​based on the first prediction processing and the predicted pixel values ​​based on the second prediction processing, according to the characteristics of the subject (image data). For example, by changing the mixing ratio α according to the environment (weather, sunlight intensity, time, etc.) in which the image processing apparatus 10 is used, the prediction accuracy of pixel values ​​in unsaturated areas can be improved. Furthermore, by changing the mixing ratio α according to the characteristics of the image of the subject, the prediction accuracy of pixel values ​​in unsaturated areas can be improved.

[0112] In the first predictive processing, by converging the corrected pixel values ​​in the fourth interval to the white balance value, it is possible to suppress the color deviation of the pixel-corrected image from the actual subject color. As a result, when the pixel values ​​of the image data containing the corrected pixel values ​​are compressed and displayed on the display device 12, an image without any sense of incongruity can be displayed.

[0113] (Second Implementation)

[0114] Figure 12 This illustrates an example of image processing performed by the image processing apparatus of the second embodiment. The image processing apparatus 10 and the image processing system 100 including the image processing apparatus 10 of this embodiment are... Figures 1 to 4 It has the same structure and can also be mounted on the mobile body 200.

[0115] In the first embodiment, image processing was described when the first to fourth intervals were arranged sequentially from left to right of the image IMG. However, in actual subjects, such as... Figure 12 As shown, the positions, sizes, and shapes of the intervals from the first to the fourth vary considerably. Furthermore, there are multiple directions for the lines that cross the interval boundaries.

[0116] Therefore, in this embodiment, the image processing apparatus 10 predicts pixel values ​​exceeding the upper limit value in each of the multiple scanning directions that cross the boundaries of the intervals. To determine the boundaries of the intervals, the image processing apparatus 10 acquires the distribution of the first to fourth intervals in the image IMG (in...) while simultaneously capturing the subject. Figure 12 (In the image, the dotted line is circular). By obtaining the distribution, the image processing device 10 is able to detect pixels whose pixel values ​​exceed the upper limit for each color.

[0117] Then, the image processing device 10 determines multiple arrangement directions (scanning directions) for predicting pixel values ​​for each pixel that exceeds the upper limit value included in the second to fourth intervals. Figure 12 In the example shown, although eight scan directions are represented toward the pixels of interest contained in the fourth interval and indicated by black circles, the number of scan directions is not limited to eight.

[0118] Next, for each scanning direction, the image processing apparatus 10 calculates approximate parameters per pixel using at least one of the first prediction processing (prediction based on different color components) and the second prediction processing (prediction based on unsaturated base color components) described in the first embodiment. Using the parameters calculated in each scanning direction, the image processing apparatus 10 predicts pixel values ​​for each color that exceed the upper limit value.

[0119] Then, the image processing apparatus 10 calculates the predicted pixel value of the pixel of interest by weighted averaging the pixel values ​​of each color predicted in multiple scanning directions, based on the distance to the saturation boundary, which serves as the boundary between the unsaturated and saturated regions. Figure 12 In this context, the fourth interval represents one pixel of interest, but the image processing device 10 sequentially sets all pixels contained in the second, third, and fourth intervals as pixels of interest, and obtains pixel values ​​for colors exceeding the upper limit value for each pixel. Then, the image processing device 10 predicts pixel values ​​in multiple scanning directions for each pixel of interest and calculates a weighted average. Additionally, as... Figure 11 The image processing device 10 can compress the obtained predicted pixel values.

[0120] As described above, in this embodiment, even when the pixel value of the subject acquired by the imaging device 19 exceeds the upper limit of the pixel value that the imaging device 19 can acquire, a correct pixel value can still be predicted. Furthermore, in this embodiment, a correct pixel value can be predicted regardless of the position of a bright subject. In this case, by weighting the pixel values ​​predicted in multiple scanning directions based on their distance to the saturation boundary, the accuracy of the predicted pixel value can be improved compared to predicting the pixel value in a single scanning direction.

[0121] (Third Implementation)

[0122] Figure 13 This illustrates an example of image processing performed by the image processing apparatus of the third embodiment. Detailed descriptions of elements identical to those in the above embodiments are omitted. The image processing apparatus 10 and the image processing system 100 including the image processing apparatus 10 of this embodiment are... Figures 1 to 4 It has the same structure and can also be mounted on the mobile body 200.

[0123] In the first and second embodiments, the scanning direction for predicting pixel values ​​is one-dimensional (linear). In this case, artifacts (noise) may appear along the line depending on the scanning direction. Therefore, in this embodiment, the image processing apparatus 10 does not perform scanning direction-dependent processing, but performs filtering operations on the surrounding pixels of the pixel of interest whose pixel value is being predicted. As a result, scanning direction-dependent processing can be eliminated, and the generation of artifacts can be suppressed.

[0124] In addition, Figure 13 The example shown illustrates obtaining the pixel values ​​of the pixels of interest included in the second interval. However, the image processing apparatus 10 sequentially sets all pixels included in the second, third, and fourth intervals as pixels of interest, and predicts the pixel values ​​of colors exceeding the upper limit value for each pixel. Furthermore, as in... Figure 11 As described above, the image processing device 10 can compress the obtained predicted pixel values.

[0125] Figure 13 The symbols tx and ty in the image processing apparatus represent the range of the surrounding pixel region of the pixel included in the prediction of the pixel value of the pixel of interest. The image processing apparatus 10 performs an extraction process to extract the boundary region from the surrounding pixel region containing the pixel of interest. Here, the boundary region is a region adjacent to the boundary of the saturated region in the unsaturated region of a pixel with the same color as the pixel of interest, and containing a predetermined number of pixels (boundary pixels) in the direction toward the unsaturated region. The pixel value of the boundary pixel is also referred to as the boundary pixel value.

[0126] The image processing device 10 uses the pixel values ​​of the boundary pixels contained in the surrounding pixel region to predict the pixel value of the pixel of interest. For example, in Figure 13 In the first interval, the pixel value of the green G focus pixel in the second interval is predicted by using the pixel value of the boundary pixel of the boundary region adjacent to the boundary of the second interval where the G value is unsaturated.

[0127] Then, the image processing apparatus 10 calculates parameters for an approximation for each pixel based on multiple boundary pixel values, using at least one of a first prediction process (prediction based on different color components) and a second prediction process (prediction based on unsaturated base color components). Using the calculated parameters, the image processing apparatus 10 predicts pixel values ​​for each color exceeding the upper limit value.

[0128] Furthermore, the image processing apparatus 10 performs a weighted average of the predicted pixel values ​​for each color based on the distance from the boundary pixel to the pixel of interest, and determines the predicted pixel value for colors that exceed the upper limit value of the pixel of interest. Using Figure 14 and Figure 15 This section describes an example of calculating the predicted pixel values ​​based on the first and second prediction processes.

[0129] Figure 14 This indicates that the first prediction process (based on predictions of different color components) is used. Figure 13 The example shown is an approximation for the prediction of pixel values. Figure 14 This represents an example of calculating the predicted pixel value for pixel G, but the predicted pixel values ​​for pixels R and B can also be calculated in the same way.

[0130] For example, with Figure 8 Similarly, the first term of equation (2), as well as equations (5-1), (5-2), and (5-3), can be considered to predict the coefficient a from the unsaturated pixels around the pixel of interest. R a B The method. As mentioned above, coefficient a R The coefficient a is calculated from the ratio of G to R. B It is calculated from the ratio of G to B, but since there are multiple unsaturated pixels, it is preferable to calculate the weighted average corresponding to the distance.

[0131] The function V(tx,ty) shown in equation (7-1) is used in term C of the approximation (7-2) for calculating the predicted GR ratio. The function V(tx,ty) is used in... Figure 13 The GR ratio is used in the pixels of the surrounding pixel region, returning "1" for unsaturated pixels (including boundary pixels) and "0" for saturated pixels. Thus, in approximation (7-2), saturated pixels can be masked, and only unsaturated pixels are used to predict the GR ratio.

[0132] The approximate formula (7-2) for calculating the predicted GR ratio represents the... Figure 13The operation is performed on all surrounding pixels within the surrounding pixel region, and the sum is calculated. The term B in approximation (7-2) represents the weight of the distance from the pixel of interest to the surrounding pixels of the saturation boundary, using a function that decreases according to the distance weight. In this embodiment, the same formula for the distance weight used in bidirectional filters, etc., is used; however, other mathematical formulas can be used as long as the formula decreases according to the distance weight.

[0133] Term D in approximation (7-2) is used to make approximation (7-2) converge to a pre-set fixed pixel value when there is no saturation boundary in the surrounding pixel region. When all pixels in the surrounding pixel region are saturated pixels and there is no saturation boundary, term C is "0". In this case, the results of terms A, B, and C in approximation (7-2) are "0", and term E is also "0". Therefore, the predicted value of the pixel of interest predicted by approximation (7-2) becomes a fixed value "C". The fixed value C is preferably set to... Figure 9 The white balance value WB is explained in the text. By setting the parameter of approximation (7-2) to the R component, the predicted GB ratio can be calculated.

[0134] Alternatively, the coefficient 'a' can also be calculated directly using the least squares method. In the first term of equation (2), since the coefficient 'a' is the variable we want to find, we can obtain the coefficient 'a' that minimizes the difference between the predicted pixel values ​​G^ from the pixels in the surrounding pixel region. Specifically, we only need to calculate the coefficient 'a' that minimizes the cost function J shown in equation (7-3). Equation (7-3) is calculated using the general least squares method, which can be performed at high speed by utilizing the method used in guided filters.

[0135] In equation (7-3), term A represents the prediction error of the G value in unsaturated pixels, which is the equation we want to minimize. However, as mentioned above, for example, when all pixels in the surrounding pixel area are saturated, the function V(tx,ty) in equation (7-1) is all "0", and a solution cannot be found.

[0136] To avoid this problem, terms B and C are defined in equation (7-3). Terms B and C in equation (7-3) are regularization terms in the least squares method. Since the cost function J increases the further the coefficient a is from the white balance value WB, the coefficient a is optimized to a value close to the white balance value WB. Thus, when the function V(tx,ty) is all "0", the coefficient a can be made to converge to the white balance value WB. The symbol k in terms B and C of equation (7-3) indicates the strength of the constraint condition.

[0137] Furthermore, through formula (7-4), dynamic calculation can be performed. Figure 14 The first prediction process shown (prediction based on different color components) and Figure 15The appropriate value of the mixing rate α for the second prediction process shown (based on unsaturated authenticity prediction). For example, by integrating... Figure 8 Using the first and second terms of equation (2) as the optimization target, the coefficients a and b shown in equation (7-4) are calculated using equation (7-3), thus eliminating the need to determine the mixing ratio α. Here, the following will be used... Figure 15 The approximation (8-3) pre-calculates F(x,y,G) of formula (7-4).

[0138] Figure 15 This indicates that the second prediction process (prediction based on unsaturated chromaticity) is used. Figure 13 The example shown is an approximation for the prediction of pixel values. Figure 15 This example illustrates the calculation of the predicted pixel value for pixel G, but the predicted pixel values ​​for pixels R and B can also be calculated in the same way. Figure 15 In this method, the boundary pixels in the surrounding pixel region are used to predict the pixel value of the pixel of interest.

[0139] Typically, since there are multiple boundary pixels in the surrounding pixel region, the predicted pixel value is obtained by weighting the average value according to the distance from the saturated pixel. Alternatively, in the case where there are no boundaries within the surrounding pixel region, formula (8-2) is used to make the predicted pixel value a fixed value.

[0140] Equation (8-1) is an approximate calculation of the differential and is computed for use in linear prediction. The function W(tx,ty) shown in equation (8-2), like equation (7-1), returns "1" for unsaturated pixels (including boundary pixels) and "0" for saturated pixels. By placing the function V(tx,ty) in term C of the approximation (8-3), pixels other than boundary pixels can be masked, allowing pixel values ​​to be predicted only from boundary pixels.

[0141] The approximate formula (8-3) for calculating the predicted value F(x, y, G) of G represents the... Figure 13 The operation is performed on all pixels within the surrounding pixel region, and the sum is calculated. Term A in approximation (8-3) represents the linear prediction value of the pixel of interest relative to the boundary pixels. Term A in approximation (8-3) is... Figure 10 The 1-dimensional linear prediction shown is expanded into a 2-dimensional formula. Alternatively, instead of using a 1-dimensional formula for approximation as in the first embodiment, a multi-dimensional formula, such as a 1-dimensional formula, can be used for approximation.

[0142] In approximation (8-3), term B represents the weight of the distance from the pixel of interest to the boundary pixel, using a function that decreases according to the distance weight. In this embodiment, the same formula for the distance weight used in bidirectional filters, etc., can also be used. Alternatively, other mathematical formulas can be used as long as the formula decreases according to the distance weight.

[0143] The term D in approximation (8-3), like the term D in approximation (7-2), is used to converge approximation (8-3) to a fixed value when there are no boundary pixels in the surrounding pixel region. The fixed value C is preferably set to the white balance value WB. The term E (denominator) in approximation (8-3) is a normalization term used for weighted averaging.

[0144] In this embodiment, the same effects as those described above can be achieved. Furthermore, in this embodiment, the image processing apparatus 10 uses boundary pixels extracted from the pixel values ​​of pixels in the surrounding pixel region of the pixel of interest whose pixel value is being predicted to predict the pixel value, and calculates the predicted pixel value by performing a weighted average of the predicted pixel values ​​based on distance. This eliminates processing dependent on the scanning direction and suppresses the generation of artifacts. Additionally, when the surrounding pixel region does not contain boundary pixels, by setting the predicted pixel value to a pre-set fixed pixel value, the pixel value of the saturated region can be correctly predicted even when boundary pixels cannot be extracted.

[0145] Furthermore, for example, when the image processing device 10 predicts pixel values ​​for the currently interested image, it performs approximate calculations after acquiring the image. Therefore, time is required from the time the image is acquired until the pixel value prediction is completed, or from the time the image is acquired until the image is displayed on a display device or the like based on the predicted pixel values. Additionally, for example, as... Figure 5 As shown, when the camera device 19 acquires an image of the subject through raster scanning, it may sometimes be impossible to calculate the parameters of the approximate formula used to calculate the predicted pixel value before acquiring an image of the entire field of view.

[0146] Therefore, the image processing apparatus 10 of the first to third embodiments described above can also use a recently acquired image or an image acquired earlier than the currently acquired image to calculate the approximate parameters. Then, the image processing apparatus 10 uses the calculated parameters to predict the pixel values ​​in the currently acquired image. As a result, the prediction processing of pixel values ​​for the currently acquired image can be performed quickly, and real-time performance can be improved. In particular, by applying it to fields that require real-time image processing, such as the camera device 19 mounted on the mobile body 200, image recognition performance can be improved.

[0147] The present invention has been described above based on various embodiments, but the present invention is not limited to the elements shown in the above embodiments. Modifications can be made to these aspects without departing from the spirit of the invention, and appropriate provisions can be made according to their application.

[0148] [Symbol Explanation]

[0149] 10 Image processing device

[0150] 10a Acquisition Department

[0151] 10b Extraction Section

[0152] 10c Forecasting Department

[0153] 10d output unit

[0154] 11 Information processing device

[0155] 12 Display devices

[0156] 14 Wireless communication devices

[0157] 15 Sensors

[0158] 16. Drive unit

[0159] 17. Lamp installation

[0160] 18 Navigation devices

[0161] 19 (19A, 19B, 19C, 19D, 19E) camera devices

[0162] 100 Image Processing System

[0163] 200 moving bodies

[0164] BUS

[0165] S stands for Sun.

Claims

1. An image processing apparatus, comprising: The extraction unit extracts unsaturated regions with unsaturated pixel values ​​and saturated regions with saturated pixel values ​​from image data representing an image containing multiple pixels acquired by the imaging device; and The prediction unit predicts the pixel value of the pixel of interest in the saturated region based on the pixel values ​​of the boundary pixels in the unsaturated region that are boundary regions with the saturated region. The image data is obtained for each pixel of multiple colors. The extraction unit extracts the unsaturated region and the saturated region from each of the multiple colors of the image data. The prediction unit performs a first prediction process, which predicts the pixel value of the saturated region of the color of interest based on the ratio of the pixel value of the color of interest (which is any one of the plurality of colors) in the boundary region to the pixel value of the other unsaturated colors.

2. The image processing apparatus according to claim 1, wherein, The prediction unit changes the ratio P1 / P2 between the pixel value P1 of the color of interest and the pixel value P2 of other unsaturated colors, such that at any pixel position in the fully saturated region where all pixel values ​​of the plurality of colors are saturated, the pixel value converges to the white balance value of the other colors.

3. The image processing apparatus according to claim 1, wherein, The prediction unit performs a second prediction process, which predicts the pixel value of the pixel of interest based on the amount of change in the pixel values ​​of a plurality of boundary pixels whose color is the same as that of the pixel whose pixel value is to be predicted, along the direction toward the saturation region.

4. The image processing apparatus according to claim 1, wherein, The image data is obtained for each pixel of multiple colors. The extraction unit extracts the unsaturated region and the saturated region from each of the multiple colors of the image data. The prediction unit performs: The first prediction process predicts the pixel value of the saturated region of the color of interest based on the ratio of the pixel value of the color of interest, which is any one of the plurality of colors, in the boundary region to the pixel value of the other unsaturated colors. as well as The second prediction process predicts the pixel value of the pixel of interest based on the amount of change in pixel values ​​of a plurality of boundary pixels with the same color as the pixel whose pixel value is to be predicted, along a direction toward the saturation region. The pixel values ​​predicted in the first prediction process are mixed with the pixel values ​​predicted in the second prediction process at a predetermined ratio.

5. The image processing apparatus according to any one of claims 1 to 4, wherein, The prediction unit predicts the pixel value of the pixel of interest by performing a weighted average of multiple pixel values ​​of the pixel of interest predicted based on the pixel values ​​of the multiple boundary pixels, according to the distance from the multiple boundary pixels to the pixel of interest.

6. The image processing apparatus according to claim 5, wherein, The extraction unit extracts the boundary region from the surrounding pixel region containing the pixel of interest. The prediction unit predicts multiple pixel values ​​of the pixel of interest based on the pixel values ​​of multiple boundary pixels of the boundary region of the surrounding pixel region.

7. The image processing apparatus according to claim 6, wherein, If the boundary region is not included in the surrounding pixel region, the prediction unit uses a preset pixel value as the predicted value of the pixel of interest.

8. The image processing apparatus according to any one of claims 1 to 4, wherein, After predicting the pixel values ​​of the saturated region in the image data, the pixel values ​​of each pixel in the image data are compressed so that the maximum value of the predicted pixel values ​​converges within the pixel values ​​of the unsaturated region.

9. The image processing apparatus according to any one of claims 1 to 4, wherein, The prediction unit performs: Processing of calculating the parameters of the approximation used in the prediction of the pixel value of the pixel of interest based on the extraction result of the extraction unit using image data pre-acquired by the camera device; and The processing of the pixel value of the pixel of interest in the image data acquired by the camera device is predicted in real time using the calculated parameters.

10. An image processing method, implemented as follows: Extraction processing involves extracting unsaturated regions with unsaturated pixel values ​​and saturated regions with saturated pixel values ​​from image data representing an image containing multiple pixels acquired by a camera device; and The prediction process involves predicting the pixel value of the pixel of interest in the saturated region based on the pixel values ​​of the boundary pixels in the unsaturated region that border the saturated region. The image data is obtained for each pixel of multiple colors. In the extraction process, the unsaturated regions and the saturated regions are extracted from each of the multiple colors of the image data. In the prediction process, a first prediction process is performed, which predicts the pixel value of the saturated region of the color of interest based on the ratio of the pixel value of the color of interest, which is any one of the plurality of colors, in the boundary region to the pixel value of the other unsaturated colors.

Images