Information processing device and information processing method

Abstract

The present invention suitably generates shading correction value data.　An applicable shading correction value data generation method for generating applicable shading correction value data to be applied to captured image data is determined by an applicable shading correction value data generation method determination unit. In this case, the determination is made in accordance with which correction model data is present among a plurality of correction model data, each of which includes a temporary shading correction value for each of a plurality of areas in the frame of captured image data. A shading correction value for each of a plurality of areas having a prescribed size smaller than that of a plurality of areas in correction model data is included in the applicable shading correction value data.

Description

[Technical field]

[0001] The present technology relates to an information processing device and an information processing method, and more particularly to an information processing device and an information processing method related to generation of shading correction data. [Background technology]

[0002] Conventionally, several correction model data related to shading correction are known. Shading (Vignetting) is a phenomenon in which the brightness of an image becomes uneven depending on the position on the image plane due to the peripheral illumination characteristics of the lens or vignetting of light rays in the imaging device. When this phenomenon is pronounced, the corners of the image become dark, and the impression of the photograph becomes unnatural. In shading correction, in order to make the brightness of the image uniform, a process of changing the brightness appropriately according to the position on the image plane (for example, applying gain) is performed with reference to the correction model data.

[0003] The correction model data includes shading correction values ​​for each of a plurality of areas in a frame of captured image data. Representative examples of correction model data include image height model correction data (e.g., see Patent Document 1) and mesh model correction data (e.g., see Patent Document 2), but there is a demand for better generation of shading correction value data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-078425 A [Patent Document 2] JP 2011-095933 A Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present technology is to generate shading correction value data satisfactorily. [Means for solving the problem]

[0006] The concept of this technology is as follows: an applied shading correction value data generation method determination unit that determines an applied shading correction value data generation method for generating applied shading correction value data including shading correction values ​​for a plurality of areas of a predetermined size smaller than the plurality of areas in the correction model data, to be applied to the captured image data, according to which correction model data is included among a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of the captured image data; Located in an information processing device.

[0007] In the present technology, an applied shading correction value data generation method determination unit determines an applied shading correction value data generation method for generating applied shading correction value data to be applied to captured image data. In this case, the applied shading correction value data generation method is determined depending on which correction model data is included among a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of the captured image data. The applied shading correction value data includes shading correction values ​​for each of a plurality of areas having a predetermined size smaller than the plurality of areas in the correction model data.

[0008] For example, the applied shading correction value data generation method determination unit may determine the applied shading correction value data generation method in accordance with the number of correction model data included among a plurality of correction model data.

[0009] In this case, for example, when there is only one correction model data among the multiple correction model data, the applied shading correction value data generation method determination unit may be configured to determine the applied shading correction value data generation method to be a single correction model data utilization generation method based on only the one correction model data among the multiple correction model data. For example, this single correction model data utilization generation method may be a method in which data of shading correction values ​​for each of multiple areas of a predetermined size based on the one correction model data is set as the applied shading correction value data.

[0010] In this case, for example, when there are first correction model data and second correction model data among the multiple correction model data, the applicable shading correction value data generation method determination unit may be configured to set the applicable shading correction value data generation method to a multiple correction model data utilization generation method based on both the first correction model data and the second correction model data.

[0011] For example, the image forming apparatus may include an applied shading correction value data generating unit that generates applied shading correction value data based on an applied shading correction value data generating method.

[0012] Also, for example, an association unit may be provided that associates the applied shading correction value data generated by the applied shading correction value data generation unit with the captured image data.

[0013] Also, for example, an associating unit may be provided that associates data related to the determination of the applied shading correction value data generation method or the result of the determination with the captured image data. Here, the data related to the determination of the applied shading correction value data generation method is, for example, correction model data. Also, the data related to the result of the determination of the applied shading correction value data generation method is predetermined area correction data for each of multiple areas of a predetermined size calculated from the correction model data, difference data, applied shading correction value data, data indicating shading correction values ​​for each of multiple pixels based on the applied shading correction value data, data indicating the applied shading correction value data generation method, etc.

[0014] Also, for example, an image processing unit may be provided that performs shading correction of captured image data using shading correction values ​​for each of multiple pixels based on the applied shading correction value data generated by the applied shading correction value data generating unit. In this case, for example, an association unit may be provided that associates the applied shading correction value data generated by the applied shading correction value data generating unit with corrected captured image data generated by the image processing unit performing shading correction on the captured image data.

[0015] In this way, in the present technology, an applicable shading correction value data generation method for generating applicable shading correction value data to be applied to the captured image data is determined depending on which of a plurality of correction model data is present, each of which includes provisional shading correction values ​​for a plurality of areas within a frame of the captured image data, and thus shading correction value data can be generated satisfactorily.

[0016] Another concept of the present technology is an applied shading correction value data generating unit that generates applied shading correction value data including shading correction values ​​for a plurality of areas of a predetermined size smaller than the plurality of areas in the correction model data, based on a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of the captured image data, to be applied to the captured image data; Located in an information processing device.

[0017] In the present technology, a shading correction value data generating unit generates applied shading correction value data to be applied to the captured image data based on a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of the captured image data. The applied shading correction value data includes shading correction values ​​for each of a plurality of areas having a predetermined size smaller than the plurality of areas in the correction model data.

[0018] For example, the plurality of correction model data may include first correction model data and second correction model data, and the first correction model data may have a higher degree of freedom in setting the shading correction value for each area than the second correction model data, where the degree of freedom means the number of data that can freely take values.

[0019] Also, for example, the plurality of correction model data may include first correction model data and second correction model data, and the first correction model data may have a lower correction resolution for the area than the second correction model data.

[0020] Also, for example, the multiple correction model data may include first correction model data and second correction model data, and the applied shading correction value data generation unit may generate the applied shading correction value data based on correction value difference data for each specified area, which is data of difference values ​​of shading correction values ​​for each of multiple areas of a specified size generated based on the first correction model data and the second correction model data.

[0021] In this case, for example, the applied shading correction value data generation unit may be configured to generate the applied shading correction value data based on first correction value data for each specified area, which is data on shading correction values ​​for each of multiple areas of a specified size based on the first correction model data, and correction value difference data for each specified area.

[0022] Also, for example, the multiple correction model data may include first correction model data and second correction model data, and the applied shading correction value data generation unit may generate the applied shading correction value data based on a magnitude of a difference value of shading correction values ​​for each predetermined area between first correction value data for each predetermined area, which is data of shading correction values ​​for each predetermined area based on the first correction model data, and second correction value data for each predetermined area, which is data of shading correction values ​​for each predetermined area based on the second correction model data.

[0023] Also, for example, the applied shading correction value data generating unit may generate the applied shading correction value data according to imager shift information related to a shift of an imager. In this case, for example, an extrapolation unit may be provided that extrapolates a provisional shading correction value to one or more areas where a provisional shading correction value is missing in a plurality of types of correction model data, and the applied shading correction value data generating unit may generate the applied shading correction value data based on the number of pieces of data of the provisional shading correction value that the extrapolation unit extrapolates to each of the plurality of correction model data according to the imager shift information.

[0024] In this way, in the present technology, applicable shading correction value data to be applied to the captured image data is generated based on multiple correction model data, each of which includes provisional shading correction values ​​for multiple areas within a frame of the captured image data, and the shading correction value data can be generated effectively. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing an example of the configuration of an imaging system according to a first embodiment. [Diagram 2] 11A and 11B are diagrams for explaining image height model correction data and shading correction based thereon. [Diagram 3] 11 is a flowchart illustrating an example of a shading correction process using image height model correction data. [Figure 4] 11A and 11B are diagrams for explaining mesh model correction data and shading correction based thereon. [Diagram 5] 13 is a flowchart showing an example of shading correction processing using mesh model correction data. [Figure 6] 13A and 13B are diagrams showing the results of a comparison of a case in which image height model correction data is used and a case in which mesh model correction data is used with respect to a plurality of items. [Figure 7] 13A to 13C are diagrams illustrating an example of image height model correction data of 16 points of correction data and an example of mesh model correction data of 16 points of correction data. [Figure 8] FIG. 13 is a diagram showing an example of a process for generating applied shading correction value data by a first method (1) in a case where both image height model correction data and mesh model correction data are available. [Figure 9] 9 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. 8. [Figure 10] FIG. 13 is a diagram showing an example of a process for generating applied shading correction value data by a first method (3) in the case where both image height model correction data and mesh model correction data are available. [Figure 11] 11 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. [Figure 12] 13 is a diagram showing an example of a process of generating applied shading correction value data by a second method in a case where both image height model correction data and mesh model correction data are present. FIG. [Figure 13]13 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. 12. [Figure 14] FIG. 13 is a diagram showing an example of a process for generating applied shading correction value data by a third method (1) in a case where both image height model correction data and mesh model correction data are available. [Figure 15] 11 is a diagram for explaining extrapolation of correction data (provisional shading correction values). FIG. [Figure 16] 13A and 13B are diagrams illustrating an example of determination of an extrapolation number between an image height model and a mesh model. [Figure 17] 15 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. 14. [Figure 18] 13 is a diagram for explaining an example in which the number of pieces of interpolated data (extrapolated data) generated using extrapolated correction data (temporary shading correction values) is determined as an extrapolation number. FIG. [Figure 19] FIG. 13 is a diagram showing an example of a process for generating applied shading correction value data by a third method (3) in the case where both image height model correction data and mesh model correction data are available. [Figure 20] 20 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. 19. [Figure 21] FIG. 13 is a diagram showing an example of a process for generating applied shading correction value data by a third method (4) in the case where both image height model correction data and mesh model correction data are available. [Figure 22] 13 is a diagram for explaining selection of one of the interpolation data for each interpolation position in accordance with attributes of the interpolation data of the image height model and the mesh model. FIG. [Figure 23] 22 is a flowchart showing an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in FIG. 21. [Figure 24] 1 is a flowchart illustrating an example of a processing procedure relating to shading correction in an imaging system. [Diagram 25] FIG. 11 is a diagram illustrating an example of the configuration of an imaging system 10A according to a second embodiment. [Figure 26] FIG. 1 is a diagram illustrating an example of a detailed configuration of an image signal processor (ISP) and a cloud server. [Figure 27] FIG. 11 is a diagram illustrating another example of a detailed configuration of an image signal processor (ISP) and a cloud server. [Figure 28] 28 is a flowchart showing an example of a processing procedure of an image signal processor (ISP) and a cloud server corresponding to the configuration example of FIG. 27. [Figure 29] 6A to 6C are diagrams illustrating an example of a correction amount of an image height model, a correction amount of a mesh model, and a correction amount of a difference (residual). [Diagram 30] FIG. 13 is a diagram showing a list of deformation correspondence between an image height model and a mesh model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, modes for carrying out the invention (hereinafter, referred to as "embodiments") will be described. The description will be made in the following order. 1. First embodiment 2. Second embodiment 3. Variations

[0027] <1. First embodiment> [Example of imaging system configuration] 1 shows an example of the configuration of an imaging system 10 according to a first embodiment. The imaging system 10 includes a lens system (optical system) 101, an imager 102, an image signal processor (ISP) 103, a lens driver 104, a vibration gyro 105, and a flash memory 106.

[0028] The lens system 101 includes lenses such as a cover lens, a zoom lens, and a focus lens, an aperture mechanism, and the like. Light (incident light) from a subject is guided by the lens system 101 and collected on an imager 102. In this embodiment, the lens system 101 also includes an image height table 101a for shading correction. Image height model correction data is written in the image height table 101a for shading correction as correction model data. The image height model correction data includes shading correction values ​​(provisional shading correction values) that are gain values ​​corresponding to a plurality of image heights.

[0029] This image height model correction data is obtained from lens design data and does not include shading variations (center variations, shape variations). Here, center variations occur when the center of the lens (center of the optical axis) is misaligned with the center of the imager (image sensor). This misalignment is due to mechanical errors in mounting the imager, lens, etc.

[0030] The imager 102 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type imager. The imager 102 outputs an imaging signal as digital data to a downstream image signal processor 103. In this embodiment, the optical image stabilization mechanism is a mechanism that corrects image shaking by moving the imager 102.

[0031] The image signal processor 103 is configured as, for example, a system-on-a-chip (SoC). The image signal processor 103 includes a camera control unit 131, a pre-processing unit 132, a synchronization unit 133, a YC generation unit 134, a post-processing unit 135, a JPEG (Joint Photographic Experts Group) generation unit 136, an output interface 137, a RAW generation unit 138, and an output interface 139.

[0032] The camera control unit 131 is configured by a microcomputer equipped with a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random access memory). The camera control unit 131 controls the entire imaging system 10 by executing a program recorded in the ROM or the flash memory 106.

[0033] For example, the camera control unit 131 controls the shutter speed of the imager 102, various signal processing operations in the image signal processor 103, operations such as zoom, focus, and aperture adjustment in the lens system 101, and image stabilization operations that shift the imager 102 based on the output of the vibration gyroscope 105.

[0034] The pre-processing unit 132 performs processes such as black level adjustment, white balance gain multiplication, pixel defect correction as RAW correction processing, color mixing correction and flicker correction, shading correction, and noise reduction on the RAW data output from the imager 102. The shading correction is performed based on the applied shading correction value data sent from the camera control unit 131. In this case, the pre-processing unit 132 can perform the shading correction well based on the applied shading correction value data sent from the camera control unit 131.

[0035] As described above, the lens system 101 includes the image height table 101a for shading correction, while the flash memory 106 includes the mesh table 106a for shading correction. Mesh model correction data is written in the mesh table 106a for shading correction as correction model data. This mesh model correction data includes shading correction values ​​(provisional shading correction values) that are gain values ​​corresponding to a plurality of mesh positions in the image area. This mesh model correction data is obtained from actual measurement data of the imager output, and includes shading variations (center variations, shape variations).

[0036] The synchronization unit 133 performs color separation processing on the RAW data output from the pre-processing unit 132 so that image data for each pixel has all color components of R (red), G (green), and B (blue). For example, in the case of the imager 102 using a Bayer array color filter, demosaic processing is performed as the color separation processing.

[0037] The YC generation unit 134 performs color gradation reproduction processing and gamma processing on the R, G, B image data (three primary color data) output from the synchronization unit 133, and then converts this R, G, B image data into a luminance signal (Y) and color difference signals (Cb, Cr) according to a predetermined calculation formula. The post-processing unit 135 performs sharpness processing and the like on the YC signal consisting of the luminance signal (Y) and color difference signals (Cb, Cr) output from the YC generation unit 134.

[0038] The JPEG generation unit 136 generates a JPEG file based on the YC signal output from the post-processing unit 135, and adds metadata associated with the JPEG file. The metadata includes data such as white balance gain and ISO sensitivity, as well as shading correction value data applied in the shading correction process (applied shading correction value data), and data related to the determination of a method for generating the shading correction value data or the result of the determination.

[0039] In this embodiment, the JPEG generating unit 136 constitutes an associating unit that associates the applied shading correction value data, and further data related to the determination or the result of the determination of the generation method of the shading correction value data, with the captured image data. The RAW generating unit 138, which will be described later, also constitutes a similar associating unit. In this way, the JPEG generating unit 136 associates the shading correction value data (applied shading correction value data) and further data related to the determination or the result of the determination of the generation method of the shading correction value data with the JPEG file, so that the associated data can be easily and appropriately used in subsequent processing. For example, the user can check the shading correction value data used for shading correction.

[0040] In this embodiment, an example of generating a JPEG file is shown, but it is also possible to generate files in other formats, such as TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF (High Efficiency Image File Format), etc.

[0041] The output interface 137 sends the JPEG file with the added metadata obtained by the JPEG generating unit 136 to an output destination device. The output destination device may be, for example, a recording device or a communication device.

[0042] The recording device records JPEG files to which metadata has been added. This recording device may be configured, for example, as a flash memory and its write / read circuit built into the imaging system 10, or may be configured as a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging system 10, such as a memory card (such as a portable flash memory), or may be realized as a HDD (hard disk drive) built into the imaging system 10.

[0043] The communication device communicates with a server on the Internet, a PC (Personal Computer) on a wired or wireless LAN (Local Area Network), or other external devices using any communication method, and transmits a JPEG file with added metadata to the communication partner.

[0044] The RAW generation unit 138 generates a RAW data file based on the RAW data that has been subjected to the RAW correction process in the pre-processing unit 132 but not yet subjected to the shading correction. The RAW data included in this RAW data file may be either uncompressed RAW data or compressed RAW data. The RAW generation unit 138 also adds metadata to the RAW data file in association with the RAW data file, similar to that in the JPEG generation unit 136 described above. In this way, the RAW generation unit 138 associates the shading correction value data (applied shading correction value data) and further data related to the determination of the method of generating the shading correction value data or the result of the determination with the RAW data file, thereby making it possible to easily and appropriately use the associated data in subsequent processing.

[0045] The output interface unit 139 sends the RAW data file with added metadata obtained by the RAW generation unit 138 to an output device such as a recording device or a communication device, similar to the output interface unit 137 described above.

[0046] The lens driver 104 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a motor of an aperture mechanism, etc. These motor drivers apply drive currents to the corresponding motors in response to instructions from the camera control unit 131, and cause the zoom lens and focus lens to move, open and close the aperture blades of the aperture mechanism, etc.

[0047] "Generating applied shading correction value data" As described above, the shading correction process in the pre-processing unit 132 is performed based on the applied shading correction value data sent from the camera control unit 131. In this embodiment, the camera control unit 131 constitutes an applied shading correction value data generating unit.

[0048] Generation of applied shading correction value data in the camera control unit 131 will be described. The camera control unit 131 determines a method of generating applied shading correction value data depending on which correction data among the multiple correction model data is held. This allows the camera control unit 131 constituting the applied shading correction value data generating unit to generate good applied shading correction value data depending on the correction data held. Here, each correction model data has provisional shading correction values ​​for each of multiple areas within a frame of captured image data.

[0049] For example, the camera control unit 131 determines the applied shading correction value data generation method according to the number of correction model data possessed among the multiple correction model data. This makes it possible to switch the correction data generation method according to the number of correction data possessed, for example, multiple or one, and it becomes possible to appropriately generate applied shading correction value data by an optimal method. In this embodiment, the camera control unit 131 determines the applied shading correction value data generation method according to whether there are both image height model correction data and mesh model correction data, only image height model correction data, or only mesh model correction data as the model correction data. Note that if there is neither image height model correction data nor mesh model correction data, the applied shading correction value data generation method is not determined and no correction is performed.

[0050] "When only image height model correction data or mesh model correction data is available" First, a case where only image height model correction data is available or only mesh model correction data is available will be described. When only image height model correction data is available or only mesh model correction data is available, the camera control unit 131 decides on a single correction model data utilization generation method based on only that one model correction data. In this case, it is possible to generate good applicable shading correction value data by utilizing the one model correction data. When only image height model correction data is available, the camera control unit 131, for example, uses the image height model correction data as it is as the applicable shading correction value data. Also, when only mesh model correction data is available, the camera control unit 131, for example, uses the mesh model correction data as it is as the applicable shading correction value data.

[0051] "Shading correction using image height model correction data" Shading correction using image height model correction data will now be described. Image height model correction data is obtained from lens design data, and includes correction data (shading correction values) that are gain values ​​corresponding to a plurality of image heights, as shown in Figs. 2(a) to (c).

[0052] Fig. 2(a) is a diagram showing correction data (shading correction values) in the image height model correction data as viewed from above. Fig. 2(b) is a diagram showing correction data (shading correction values) in the image height model correction data as viewed from the side. Fig. 2(c) is a diagram showing correction data (shading correction values) in the image height model correction data as viewed obliquely from above.

[0053] In this way, the correction data (shading correction values) in the image height model correction data are positioned like each circle constituting a concentric circle, and the correction data on each circle has the same value. In the shading correction process using this image height model correction data, correction is performed concentrically around the optical axis center. In this case, as shown in FIG. 2(d), shading correction of a pixel of interest is performed by calculating correction data for the pixel of interest by performing interpolation (for example, linear interpolation) from correction data of two points closest to the pixel of interest on a line connecting the optical axis center (correction center) and the pixel of interest. The optical axis center is detected during the manufacturing process of the camera (imaging system) and is stored in memory for each set.

[0054] The flowchart in Fig. 3 shows an example of shading correction processing using image height model correction data. First, in step ST1, the optical axis center is set to the image height center. Next, in step ST2, the image height (distance) from the image height center of the pixel of interest is calculated. Next, in step ST3, pixel data of the pixel of interest is interpolated from the correction data (two points) closest to the image height model correction data. Next, in step ST4, image data of the pixel of interest is multiplied by the correction data of the pixel of interest.

[0055] "Shading correction using mesh model correction data" Shading correction using mesh model correction data will now be described. Mesh model correction data is obtained from actual measurement data of imager output. In this case, as shown in Fig. 4(a) and (b), it includes correction data (shading correction values) that are gain values ​​corresponding to multiple mesh positions in the image area. Fig. 4(b) is a view of the correction data (shading correction values) in the mesh model correction data viewed from diagonally above.

[0056] In this case, as shown in Fig. 4(a), the image area is provided with m x n correction points, and correction data for each correction point is detected. In this case, the correction data for each correction point is detected based on peripheral light falloff when an object with full pixel luminance saturation is photographed. For example, it is assumed that a user will send a combination of a lens and a body to a factory to obtain a mesh table, or that a mesh table will be automatically generated by the user photographing an object with full pixel luminance saturation.

[0057] In this way, the correction data (shading correction values) in the mesh model correction data are positioned in a mesh (matrix) shape. In the shading correction process using this mesh model correction data, the correction data for the pixel of interest is calculated by performing interpolation (for example, linear interpolation) from the correction data for the four points closest to the pixel of interest, as shown in FIG. 4(a).

[0058] The flowchart in Fig. 5 shows an example of shading correction processing using mesh model correction data. First, in step ST11, pixel data of a pixel of interest is interpolated from the nearest correction data (four points) of the mesh model correction data. Next, in step ST12, image data of the pixel of interest is multiplied by the correction data of the pixel of interest.

[0059] "Comparison between image height model and mesh model" Figure 6 shows the results of comparing multiple items between the case where image height model correction data is used and the case where mesh model correction data is used. The degree of freedom for correction is higher when mesh model correction data is used. In this case, shading correction using image height model correction data can accommodate central variation in shading, but since only concentric correction can be performed in the image height direction, it cannot accommodate variation in shading shape. In contrast, shading correction using mesh model correction data can accommodate both central variation and shape variation in shading.

[0060] When the data size is the same, the correction resolution is higher when the image height model correction data is used. In this case, when the image height model correction data and the mesh model correction data have the same data size, the correction density of the mesh model is smaller than that of the image height model, and therefore the correction resolution is lower.

[0061] For example, Fig. 7(a) shows an example of image height model correction data with 16 correction data points, and Fig. 7(b) shows an example of mesh model correction data with 16 correction data points. In this case, the image height model correction data has 16 correction data points in all image height directions from the center of the optical axis. In contrast, the mesh model correction data has only a rough number of correction points, four in the vertical direction and four in the horizontal direction.

[0062] Regarding calibration, it is more advantageous to use mesh model correction data. When using image height model correction data, it is necessary to detect the variation in the center of the optical axis in the manufacturing process. On the other hand, when using mesh model correction data, the correction data is obtained from the actual measurement data of the imager output, and the correction data includes variation, so no adjustment is required.

[0063] In terms of calculation costs, using mesh model correction data is advantageous because it is lower. When image height model correction data is used, two-dimensional distance calculation is required to calculate the image height, which is the distance from the optical axis center to the pixel of interest, and square root calculation is a heavy process. In contrast, when mesh model correction data is used, such two-dimensional distance calculation is not necessary and processing is lighter.

[0064] In terms of the range of correction, using the image height model correction data is more advantageous because it has a wider range. The image height model correction data has correction data within the range of the image circle, so the correction data extends beyond the effective image area (imager area). In contrast, the mesh model correction data has correction data only for the effective image area (imager area).

[0065] "When both image height model correction data and mesh model correction data are present" Next, a case where both image height model correction data and mesh model correction data are present will be described. When both image height model correction data and mesh model correction data are present, the camera control unit 131 determines to use a multiple correction model utilization generation method based on these two model correction data. In this case, it is possible to generate more accurate applied shading correction value data by utilizing both the image height model correction data and the mesh model correction data.

[0066] The camera control unit 131 generates applied shading correction value data (correction data with a predetermined resolution) based on the image height model correction data and the mesh model correction data by any one of the following first to third methods. This applied shading correction value data has the advantages of both the image height model correction data and the mesh model correction data, and is highly accurate. In this case, the mesh model correction data has a higher degree of freedom in setting shading correction values ​​for each area than the image height model correction data, and therefore this advantage can be provided. In addition, the image height model correction data has a higher area correction resolution than the mesh model correction data, and therefore this advantage can be provided.

[0067] The correction data (shading correction values) in the correction data with the predetermined resolution are arranged in a mesh (matrix) shape. Therefore, in the shading correction process, the correction data of the pixel of interest is calculated by performing interpolation (for example, linear interpolation) from the correction data of the four points nearest to the pixel of interest, in the same manner as the shading correction process using the mesh model correction data described above (see FIG. 4(a)).

[0068] "Method 1" In this first method, applied shading correction value data is generated based on correction value difference data for each specified area (difference data with a specified resolution), which is data of difference values ​​of shading correction values ​​for multiple areas of a specified size generated based on image height model correction data and mesh model correction data.

[0069] In this case, the image height model correction data and the mesh model correction data each include correction data (provisional shading correction values) for each of a plurality of areas in a frame of captured image data, but the above-mentioned area of ​​the predetermined size is smaller than the area in the image height model correction data or the mesh model correction data. In this case, since the difference data of the predetermined resolution corresponds to the shading variation (center variation, shape variation), it is possible to generate applied shading correction value data corresponding to the shading variation (center variation, shape variation).

[0070] "First Method (1)" 8 shows an example of a process for generating applied shading correction value data by the first method. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0071] From the image height model correction data (see FIG. 8(a)), interpolated data of a predetermined resolution (see FIG. 8(b)) is generated by interpolating in a mesh shape corresponding to the imager area (effective image area). This mesh-shaped interpolated data of a predetermined resolution constitutes a shading correction value for each of a plurality of areas of a predetermined size. For example, the predetermined resolution is set to a resolution desired (to be obtained) by the user. In this case, the resolution is set to a high value when high correction accuracy is required, and the resolution is set to a low value when low correction accuracy is acceptable. In the illustrated example, the interpolated data of a predetermined resolution has 1000 correction points.

[0072] There are several ways to determine the resolution, for example: (1) to determine it uniquely for the lens model connected, or (2) to determine it according to the lens parameters of the lens connected. In the case of (1), the shading shape changes depending on the lens parameters (aperture value, focus position, zoom position, focal length, etc.) even for the same lens model, but the resolution is determined, for example, taking into consideration the case where the highest correction accuracy is required for that lens. In the case of (2), for example, the resolution is determined for a representative lens parameter, and the resolution is switched when the lens parameter is switched.

[0073] Further, from the mesh model correction data (see FIG. 8(c)), interpolation data of a predetermined resolution is generated by interpolating into a mesh shape corresponding to the imager area (effective image area) (see FIG. 8(d)). This mesh-shaped interpolation data of a predetermined resolution constitutes shading correction values ​​for each of a plurality of areas of a predetermined size. In this case, the resolution of the mesh-shaped interpolation data generated from the mesh model correction data is made the same as the resolution of the mesh-shaped interpolation data generated from the image height model correction data described above. In the illustrated example, the interpolation data of a predetermined resolution has 1000 correction points.

[0074] Furthermore, division is performed for each interpolation position between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data (see FIG. 8(e)), to obtain difference data of a predetermined resolution. In this case, the interpolation data of the mesh model is divided by the interpolation data of the image height model. This difference data constitutes correction value difference data for each predetermined area, which is data of difference values ​​of shading correction values ​​for each of a plurality of areas of a predetermined size generated based on the image height model correction data and the mesh model correction data.

[0075] As described above, the image height model correction data does not include shading variations (center variations, shape variations), but the mesh model correction data includes shading variations (center variations, shape variations). Therefore, the difference data of a predetermined resolution corresponds to the shading variations (center variations, shape variations).

[0076] Next, the difference data of the predetermined resolution is smoothed by a low-pass filter to obtain smoothed difference data of the predetermined resolution (see FIG. 8(f)).

[0077] Next, the interpolation data of a predetermined resolution generated from the image height model correction data is multiplied by the smoothed difference data of a predetermined resolution for each interpolation position (see FIG. 8(g)), to obtain correction data of a predetermined resolution, that is, applied shading correction value data (see FIG. 8(h)). As described above, the difference data of a predetermined resolution corresponds to the shading variation (center variation, shape variation), and therefore the applied shading correction value data corresponds to the shading variation (center variation, shape variation).

[0078] The flowchart in Fig. 9 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 8. First, in step ST21, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the image height model correction data. Next, in step ST22, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the mesh model correction data.

[0079] Next, in step ST23, the camera control unit 131 performs division for each interpolation position between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data to obtain difference data of a predetermined resolution. Next, in step ST24, the camera control unit 131 smoothes the difference data of the predetermined resolution to obtain smoothed difference data of the predetermined resolution.

[0080] Next, in step ST25, the camera control unit 131 multiplies the interpolation data of a predetermined resolution generated from the image height model correction data by the smoothed difference data of a predetermined resolution for each interpolation position to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data).

[0081] In the process of generating applied shading correction value data shown in FIG. 8, interpolation data of a predetermined resolution generated from image height model correction data with high correction resolution is multiplied by difference data of a predetermined resolution to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data). This makes it possible to obtain highly accurate applied shading correction value data that corresponds to shading variations (center variation, shape variation).

[0082] "First Method (2)" In the generation process of the applied shading correction value data shown in Fig. 8, a smoothing process is performed on the difference data of a predetermined resolution. Since this smoothing process is also a linear process, the location where the smoothing process is performed is not limited to this position, and may be a location after the multiplication process.

[0083] "The first method (3)" 8, the process of generating the applied shading correction value data is configured to obtain difference data of a predetermined resolution by performing division between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data. However, since the interpolation process is a linear process, it is also possible to obtain difference data of a predetermined resolution by performing division between the interpolation data of a predetermined resolution generated from the image height model correction data and the mesh model correction data.

[0084] 10 shows an example of a process for generating applied shading correction value data in this case. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0085] From the image height model correction data (see FIG. 10(a)), interpolation data (see FIG. 10(b)) with a predetermined resolution is generated by interpolating in a mesh shape corresponding to the imager area (effective image area). This mesh-shaped interpolation data with a predetermined resolution constitutes shading correction values ​​for each of a plurality of areas of a predetermined size. In the illustrated example, the interpolation data with a predetermined resolution has 1000 correction points.

[0086] Furthermore, between the interpolation data of a predetermined resolution generated from the image height model correction data and the mesh-like correction data (see FIG. 10(c)), division is performed for each mesh position of the interpolation data of the image height model (see FIG. 10(d)), to obtain difference data of a predetermined resolution that is interpolated in a mesh shape (see FIG. 10(e)). This difference data of a predetermined resolution has the same resolution (1000 correction points) as the interpolation data of the image height model.

[0087] This difference data of a predetermined resolution constitutes correction value difference data for each predetermined area, which is data of difference values ​​of shading correction values ​​for each of a plurality of areas of a predetermined size generated based on the image height model correction data and the mesh model correction data, similar to the difference data of a predetermined resolution obtained by performing division in the generation process of the applied shading correction value data shown in FIG.

[0088] Furthermore, the difference data of the predetermined resolution is smoothed by a low-pass filter to obtain smoothed difference data of the predetermined resolution (see FIG. 10(f)).

[0089] In addition, the interpolation data of a predetermined resolution generated from the image height model correction data is multiplied by the smoothed difference data of a predetermined resolution for each interpolation position (see FIG. 10(g)), to obtain correction data of a predetermined resolution, i.e., applied shading correction value data (see FIG. 10(h)).

[0090] The flowchart in Fig. 11 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 10. First, in step ST31, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated in a mesh shape from the image height model correction data.

[0091] Next, in step ST32, the camera control unit 131 performs division for each mesh position in the mesh model correction data between the interpolation data of a predetermined resolution generated from the image height model correction data and the mesh model correction data, to obtain difference data of a predetermined resolution that is interpolated into a mesh shape having the same resolution as the interpolation data of the image height model.

[0092] Next, in step ST33, the camera control unit 131 smoothes the difference data of the predetermined resolution to obtain smoothed difference data of the predetermined resolution.

[0093] Next, in step ST34, the camera control unit 131 multiplies the interpolation data of a predetermined resolution generated from the image height model correction data by the smoothed difference data of a predetermined resolution for each interpolation position to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data).

[0094] 10, the interpolation data of a predetermined resolution (see FIG. 10(b)) generated from the image height model correction data and the mesh-shaped correction data (see FIG. 10(c)) are divided for each mesh position of the interpolation data of the image height model to obtain difference data of a predetermined resolution interpolated in a mesh shape having the same resolution as the interpolation data of the image height model. However, it is also possible to obtain difference data of a predetermined resolution interpolated in a mesh shape having the same resolution as the interpolation data of the image height model by dividing the interpolation data of a predetermined resolution (see FIG. 10(b)) generated from the image height model correction data and the mesh-shaped correction data (see FIG. 10(c)) for each mesh position of the mesh model correction data to obtain difference data of a predetermined resolution interpolated in a mesh shape having the same resolution as the interpolation data of the image height model.

[0095] Although not described above, in the generation process of the applied shading correction value data shown in Fig. 8 and Fig. 10, it is also possible to use the interpolation data with a predetermined resolution generated from the image height model correction data after correcting it so as to include the center variation. In this case, the image height center is shifted and corrected so as to coincide with the optical axis center.

[0096] By using the applied shading correction value data after correcting it to include the central variation in this way, the accuracy of the applied shading correction value data can be improved. In this case, since the interpolation data of a predetermined resolution generated from the image height model correction data includes the central variation in shading, the difference data of the predetermined resolution corresponds only to the shape variation in shading.

[0097] "Second Method" In this second method, applied shading correction value data is generated based on the magnitude of a difference value of shading correction values ​​for each predetermined area between first correction value data for each predetermined area, which is data of shading correction values ​​for each plurality of areas of a predetermined size based on the image height model correction data, and second correction value data for each predetermined area, which is data of shading correction values ​​for each plurality of areas of a predetermined size based on the mesh model correction data. In this case, the image height model correction data and the mesh model correction data each include correction data (tentative shading correction values) for each plurality of areas in a frame of captured image data, but the above-mentioned area of ​​the predetermined size is an area smaller than the area in the image height model correction data or the mesh model correction data.

[0098] 12 shows an example of a process for generating applied shading correction value data by the second method. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0099] From the image height model correction data (see FIG. 12(a)), interpolation data (see FIG. 12(b)) with a predetermined resolution is generated by interpolating in a mesh shape corresponding to the imager area (effective image area). This mesh-shaped interpolation data with a predetermined resolution constitutes shading correction values ​​for each of a plurality of areas of a predetermined size. In the illustrated example, the interpolation data with a predetermined resolution has 1000 correction points.

[0100] Further, from the mesh model correction data (see FIG. 12(c)), interpolation data of a predetermined resolution is generated by interpolating into a mesh shape corresponding to the imager area (effective image area) (see FIG. 12(d)). This mesh-shaped interpolation data of a predetermined resolution constitutes shading correction values ​​for each of a plurality of areas of a predetermined size. In this case, the resolution of the mesh-shaped interpolation data generated from the mesh model correction data is made the same as the resolution of the mesh-shaped interpolation data generated from the image height model correction data described above. In the illustrated example, the interpolation data of a predetermined resolution has 1000 correction points.

[0101] In addition, a process of calculating a difference between the interpolation data of a predetermined resolution generated from the image height model correction data and the mesh model correction data and converting the difference into an absolute value is performed for each interpolation position, thereby obtaining an absolute difference value (see FIG. 12(e)).

[0102] Moreover, the absolute difference values ​​for each interpolation position are normalized to obtain an α value for each interpolation position (see FIG. 12(f)). For example, the α value for each interpolation position is obtained by dividing the absolute difference value for each interpolation position by the maximum value of the absolute difference values ​​for each interpolation position. Furthermore, for example, the α value for each interpolation position is set to 0 when the absolute difference value for each interpolation position is equal to or less than a first threshold value on the lower side, set to 1 when the absolute difference value for each interpolation position is equal to or greater than a second threshold value on the upper side, and set to a value between 0 and 1 depending on the magnitude when the absolute difference value is between the first threshold value and the second threshold value.

[0103] In addition, using the α value for each interpolation position, the interpolation data generated from the image height model correction data and the interpolation data generated from the mesh model correction data are blended (α blended) for each interpolation position (see FIG. 12(g)), to obtain correction data with a predetermined resolution, that is, applied shading correction value data (see FIG. 12(h)).

[0104] Here, the alpha blending is given by, for example, the following formula (1). "A" indicates the interpolation data generated from the mesh model correction data, "B" indicates the interpolation data generated from the image height model correction data, and "out" indicates the correction data after alpha blending (shading correction value). This formula means that when the absolute difference value, and therefore the alpha value, is small, the component of the interpolation data generated from the image height model correction data is made larger, while when the absolute difference value, and therefore the alpha value, is large, the component of the interpolation data generated from the mesh model correction data is made larger. out = A × α + B × (1 - α) (1)

[0105] This is based on the assumption that when the absolute difference value is small, the interpolated data generated from the image height model correction data is close to the interpolated data generated from the mesh model correction data, and correction data (shading correction value) with higher accuracy can be obtained by increasing the components of the interpolated data generated from the image height model correction data having a high correction resolution, while when the absolute difference value is large, the interpolated data generated from the image height model correction data is not close to the interpolated data generated from the mesh model correction data, and correction data (shading correction value) with higher accuracy can be obtained by increasing the components of the interpolated data generated from the mesh model correction data including shading variation.

[0106] In addition, the correction data of a specified resolution obtained by alpha blending (applied shading correction value data) is smoothed using a low-pass filter to obtain final smoothed correction data of a specified resolution (applied shading correction value data) (see FIG. 12(i)).

[0107] The flowchart in Fig. 13 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 12. First, in step ST41, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the image height model correction data. Next, in step ST42, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the mesh model correction data.

[0108] Next, in step ST43, the camera control unit 131 calculates a difference for each interpolation position between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data, and converts the difference into an absolute value to obtain an absolute difference value for each interpolation position.

[0109] Next, in step ST44, the camera control unit 131 blends the image height model interpolation data and the mesh model interpolation data using an α value obtained by normalizing the absolute difference value for each interpolation position to obtain mesh-like correction data (applied shading correction value data) with a predetermined resolution. Next, in step ST45, the camera control unit 131 smoothes the mesh-like correction data (applied shading correction value data) with a predetermined resolution to obtain final mesh-like correction data (applied shading correction value data) with a predetermined resolution.

[0110] In the process of generating the applied shading correction value data shown in FIG. 12, the applied shading correction value data is generated based on the magnitude of the difference between the shading correction values ​​for each predetermined area between the first per-predetermined-area correction value data, which is data of shading correction values ​​for each of a plurality of areas of a predetermined size based on the image height model correction data, and the first per-predetermined-area correction value data, which is data of shading correction values ​​for each of a plurality of areas of a predetermined size based on the mesh model correction data, and it is possible to generate applied shading correction value data with higher accuracy.

[0111] 12, the process of generating the applied shading correction value data is to use the α value for each interpolation position to blend (α blend) the interpolation data generated from the image height model correction data and the interpolation data generated from the mesh model correction data for each interpolation position to obtain mesh-like correction data with a predetermined resolution (applied shading correction value data). However, it is also conceivable to select, for each interpolation position, either the mesh model or the image height model to be used as the interpolation data depending on whether the absolute value of the difference between the interpolation data of the image height model and the mesh model is equal to or greater than a predetermined value, and obtain mesh-like correction data with a predetermined resolution (applied shading correction value data).

[0112] "The third way" In the third method, the applied shading correction value data is generated in accordance with imager shift information relating to the shift of the imager 102. This makes it possible to generate suitable applied shading correction value data in accordance with the imager shift.

[0113] "The third method (1)" 14 shows an example of a process for generating applied shading correction value data by the third method. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0114] Interpolated data of a predetermined resolution is generated by performing mesh-like interpolation from the image height model correction data (see FIG. 14(a)). This mesh-like interpolated data of a predetermined resolution constitutes shading correction values ​​for multiple areas of a predetermined size. In the illustrated example, the interpolated data of a predetermined resolution has 1000 correction points.

[0115] In this case, based on the imager shift information, correction data (provisional shading correction values) are extrapolated to one or more areas where correction data (provisional shading correction values) of the image height model correction data is missing, and this extrapolated correction data (provisional shading correction values) is used to generate interpolated data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area (effective image area). Note that the imager 102 is shift-controlled for camera shake correction during the exposure period. The imager shift information described above has, for example, averaged information on the shift position of the imager 102 that shifts during the exposure period.

[0116] 15 is a diagram for explaining extrapolation. Circles indicate correction data (provisional shading correction values) present in the image height model correction data. Squares indicate extrapolated correction data (provisional shading correction values). The extrapolation of the correction data (provisional shading correction values) is performed based on an approximation function of the correction data (provisional shading correction values) present in the image height model correction data.

[0117] Further, from the mesh model correction data, interpolation data of a predetermined resolution is generated by interpolating in a mesh shape (see FIG. 14(b)). This mesh-shaped interpolation data of a predetermined resolution constitutes a shading correction value for each of a plurality of areas of a predetermined size. In this case, the resolution of the mesh-shaped interpolation data generated from the mesh model correction data is set to be the same as the resolution of the mesh-shaped interpolation data generated from the image height model correction data described above. In the illustrated example, the interpolation data of a predetermined resolution has 1000 correction points.

[0118] In this case, similarly to the case where interpolation data of a predetermined resolution is generated from the image height model correction data described above, correction data (provisional shading correction values) are extrapolated to one or more areas where correction data (provisional shading correction values) of the mesh model correction data is missing, based on the imager shift information, and this extrapolated correction data (provisional shading correction values) is used to generate interpolation data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area (effective image area).

[0119] In addition, the extrapolation numbers of the correction data (provisional shading correction values) in the image height model and the mesh model are determined based on the image height model correction data and the mesh model correction data, and further on the imager shift information (see FIG. 14(c)).

[0120] Then, depending on the determination result of the extrapolation number, either interpolation data of a predetermined resolution obtained by interpolating the image height model correction data in a mesh-like manner or interpolation data of a predetermined resolution obtained by interpolating the mesh model correction data in a mesh-like manner is selectively extracted (see FIG. 14(d)), and correction data of the predetermined resolution, that is, applied shading correction value data (see FIG. 14(e)) is obtained.

[0121] In this case, when the extrapolation number of the image height model is smaller than the extrapolation number of the mesh model, interpolation data of a predetermined resolution obtained by interpolating in a mesh shape from the image height model correction data is selected. When the extrapolation number of the image height model is larger than the extrapolation number of the mesh model, interpolation data of a predetermined resolution obtained by interpolating in a mesh shape from the mesh model correction data is selected. When the extrapolation number of the mesh model and the extrapolation number of the image height model are the same, either the interpolation data of a predetermined resolution obtained by interpolating in a mesh shape from the image height model correction data or the interpolation data of a predetermined resolution obtained by interpolating in a mesh shape from the mesh model correction data is selected, for example, by a setting made in advance.

[0122] Fig. 16 shows an example of the determination of the extrapolation numbers of the image height model and the mesh model. Fig. 16(a) shows an example of the case where there is no shift of the imager 102. In this case, in order to generate interpolation data of a predetermined resolution obtained by interpolating the image height model correction data in a mesh shape corresponding to the imager area, it is not necessary to extrapolate the correction data (provisional shading correction value), and the extrapolation number is none, that is, 0. Also, in this case, in order to generate interpolation data of a predetermined resolution obtained by interpolating the image height model correction data in a mesh shape corresponding to the imager area, it is not necessary to extrapolate the correction data (provisional shading correction value), and the extrapolation number is none, that is, 0. Therefore, in this case, it is determined that the extrapolation numbers of the image height model and the mesh model are the same, and the image height model or the mesh model is selected according to, for example, a previous setting.

[0123] 16(b) shows an example in which the imager 102 is shifted to the left. In this case, in order to generate interpolated data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area from the image height model correction data, it is necessary to extrapolate five points of correction data (provisional shading correction values). Also, in this case, in order to generate interpolated data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area from the mesh model correction data, it is necessary to extrapolate four points of correction data (provisional shading correction values). Therefore, in this case, the number of extrapolations of the mesh model is small, so the mesh model is selected.

[0124] 16(c) shows an example in which the imager 102 is shifted upward. In this case, in order to generate interpolated data of a predetermined resolution obtained by interpolating in a mesh shape corresponding to the imager area from the image height model correction data, it is necessary to extrapolate one point of correction data (provisional shading correction value). Also, in this case, in order to generate interpolated data of a predetermined resolution obtained by interpolating in a mesh shape corresponding to the imager area from the mesh model correction data, it is necessary to extrapolate five points of correction data (provisional shading correction value). Therefore, in this case, since the number of extrapolations of the image height model is small, the image height model is selected.

[0125] 16(d) shows an example in which the imager 102 is shifted diagonally to the upper left. In this case, in order to generate interpolated data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area from the image height model correction data, it is necessary to extrapolate five points of correction data (provisional shading correction values). Also, in this case, in order to generate interpolated data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area from the mesh model correction data, it is necessary to extrapolate nine points of correction data (provisional shading correction values). Therefore, in this case, since the number of extrapolations of the image height model is small, the image height model is selected.

[0126] The flowchart in Fig. 17 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 14. First, in step ST51, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the image height model correction data. Next, in step ST52, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the mesh model correction data.

[0127] Next, in step ST53, the camera control unit 131 determines the extrapolation numbers of the correction data (provisional shading correction values) in the image height model and the mesh model based on the image height model correction data, the mesh model correction data, and further the imager shift information.

[0128] Next, in step ST54, the camera control unit 131 selects, depending on the result of the determination of the extrapolation number, either interpolation data of a predetermined resolution generated from the image height model correction data or interpolation data of a predetermined resolution generated from the mesh model correction data, to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data).

[0129] In the process of generating the applied shading correction value data shown in FIG. 14, the number of correction data (tentative shading correction values) to be extrapolated when generating interpolated data of a predetermined resolution interpolated in a mesh shape corresponding to the imager area (effective image area) is determined as the extrapolation number, and the interpolated data of a predetermined resolution interpolated in a mesh shape of the image height model and the interpolated data of a predetermined resolution interpolated in a mesh shape of the mesh model, whichever has the smaller number of extrapolations, is selected to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data). As the mesh-shaped correction data of a predetermined resolution, it is possible to obtain more accurate data with less extrapolated correction data (tentative shading correction values) required when generating the interpolated data.

[0130] "The Third Method (2)" In the extrapolation number determination process in the generation process of the applied shading correction value data shown in Fig. 14 described above, the number of correction data (provisional shading correction values) to be extrapolated in generating interpolation data of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager area (effective image area) is determined as the extrapolation number. However, in this extrapolation number determination process, it is also conceivable to determine the number of interpolation data (extrapolation data) generated using the extrapolated correction data (provisional shading correction values) as the extrapolation number.

[0131] Fig. 18(a) shows an example in which the imager 102 is shifted to the left. Fig. 18(b) shows an enlarged view of a portion of Fig. 18(a), in which the extrapolated data is represented by rectangles. In this case, the extrapolated data in the image height model is both the gray rectangles and the white rectangles, and there are 108 pieces of data. In this case, the extrapolated data in the mesh model is only the gray rectangles, and there are 64 pieces of data. Therefore, in this case, the mesh model is selected because the number of extrapolations in the mesh model is small.

[0132] The number of interpolated data (extrapolated data) generated using the correction data (provisional shading correction value) thus extrapolated is determined as the extrapolation number, and the interpolated data of a predetermined resolution interpolated in a mesh shape of the image height model and the interpolated data of a predetermined resolution interpolated in a mesh shape of the mesh model, whichever has the smaller number of extrapolations, is selected to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data).As mesh-shaped correction data of a predetermined resolution, more accurate data with less extrapolated data can be obtained.

[0133] "The Third Method (3)" In the process of generating the applied shading correction value data shown in FIG. 14 described above, depending on the determination results of the extrapolation numbers of the image height model and the mesh model, the interpolation data with a predetermined resolution generated from the image height model correction data or the interpolation data with a predetermined resolution generated from the mesh model correction data is selected to obtain mesh-like correction data with a predetermined resolution (applied shading correction value data).

[0134] However, it is also possible to obtain mesh-like correction data of a predetermined resolution (applied shading correction value data) by α-blending, for each interpolation position, the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data, depending on the extrapolation numbers of the image height model and the mesh model.

[0135] 19 shows an example of a process for generating applied shading correction value data in this case. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0136] Similar to the generation process of applied shading correction value data shown in FIG. 14, interpolation data of a predetermined resolution (interpolation data of the image height model) interpolated in a mesh shape is generated from the image height model correction data (see FIG. 19(a)), and interpolation data of a predetermined resolution (interpolation data of the mesh model) interpolated in a mesh shape is generated from the mesh model correction data (see FIG. 19(b)).

[0137] Furthermore, based on the image height model correction data, the mesh model correction data, and further the imager shift information, the extrapolation number of the image height model and the extrapolation number of the mesh model are obtained, and the α value is calculated using these extrapolation numbers (see FIG. 19(c)). Here, the extrapolation number may be the number of correction data (provisional shading correction values) to be extrapolated, or may be the number of interpolation data (extrapolation data) generated using the extrapolated correction data (provisional shading correction values). The α value is given by, for example, the following formula (2). α = extrapolation number of mesh model / (extrapolation number of image height model + extrapolation number of mesh model) (2)

[0138] In addition, using the α value, the interpolated data of the image height model and the interpolated data of the mesh model are blended (α blended) for each interpolation position (see FIG. 19(d)), and correction data of a predetermined resolution, that is, applied shading correction value data (see FIG. 19(e)), is obtained.

[0139] Here, the alpha blend is given by, for example, the following formula (3): "A" indicates the interpolated data generated from the image height model correction data, "B" indicates the interpolated data generated from the mesh model correction data, and "out" indicates the correction data after alpha blending (applied shading correction value). out = A × α + B × (1 - α) (3)

[0140] This formula means that when the extrapolation number of the image height model is smaller than the extrapolation number of the mesh model and therefore the α value is large, the components of the interpolated data generated from the image height model correction data are made larger, whereas when the extrapolation number of the mesh model is smaller than the extrapolation number of the image height model and therefore the α value is small, the components of the interpolated data generated from the mesh model correction data are made larger. This is because it is assumed that mesh-shaped interpolated data of a predetermined resolution related to correction model data with a smaller number of extrapolations has higher accuracy.

[0141] The flowchart in Fig. 20 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 19. First, in step ST61, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the image height model correction data. Next, in step ST62, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the mesh model correction data.

[0142] Next, in step ST63, the camera control unit 131 calculates extrapolation numbers for the image height model and the mesh model based on the image height model correction data, the mesh model correction data, and further the imager shift information, and calculates an α value from the extrapolation numbers (see formulas (2) and (3)).

[0143] Next, in step ST64, the camera control unit 131 blends the interpolation data of the image height model and the interpolation data of the mesh model for each interpolation position using the α value to obtain mesh-like correction data (applied shading correction value data) with a predetermined resolution.

[0144] In the generation process of the applied shading correction value data shown in FIG. 19, the α value obtained based on the extrapolation number is used to α-blend the interpolation data of a predetermined resolution interpolated into a mesh shape of the image height model and the interpolation data of a predetermined resolution interpolated into a mesh shape of the mesh model to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data), and it is possible to obtain more accurate data as the mesh-shaped correction data of a predetermined resolution.

[0145] "The Third Method (4)" In the process of generating the applied shading correction value data shown in FIG. 14 described above, the interpolation data with a predetermined resolution generated from the image height model correction data or the interpolation data with a predetermined resolution generated from the mesh model correction data is selected according to the extrapolation number, to obtain mesh-like correction data with a predetermined resolution (applied shading correction value data).

[0146] In addition, in the process of generating the applied shading correction value data shown in FIG. 19, the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data are α-blended for each interpolation position in accordance with the extrapolation number, to obtain mesh-like correction data of a predetermined resolution (applied shading correction value data).

[0147] However, it is also possible to obtain mesh-like correction data (applied shading correction value data) with a predetermined resolution by selecting either the interpolation data of the image height model or the mesh model for each interpolation position according to the attributes of the interpolation data of the image height model and the mesh model at that interpolation position. This is based on the idea of ​​using, for each interpolation position, the interpolation data of the image height model or the mesh model, whichever has higher accuracy.

[0148] 21 shows an example of a process for generating applied shading correction value data in this case. In the example shown, the image height model correction data has 16 correction points, and the mesh model correction data has 100 correction points.

[0149] Similar to the generation process of applied shading correction value data shown in FIG. 14, interpolation data of a predetermined resolution (interpolation data of the image height model) interpolated in a mesh shape from the image height model correction data is generated (see FIG. 21(a)), and interpolation data of a predetermined resolution (interpolation data of the mesh model) interpolated in a mesh shape is generated from the mesh model correction data (interpolation data of the mesh model) (see FIG. 21(b)).

[0150] In addition, based on the image height model correction data, the mesh model correction data, and further the imager shift information, it is determined for each interpolation position whether to select the image height model or the mesh model interpolation data (see FIG. 20(c)).

[0151] Here, if either the interpolation data of the image height model or the mesh model is extrapolated data (interpolated data generated using extrapolated correction data (temporary shading correction value)), the one that is not the extrapolated data is selected.

[0152] Furthermore, when both the interpolation data of the image height model and the mesh model are extrapolated data, attention is paid to which extrapolation data (extrapolated correction data (temporary shading correction value)) from the no-extrapolation region is used to generate the interpolation data, and the one with the smaller number is selected. In this case, when the numbers are the same, either the interpolation data of the image height model or the mesh model is selected, for example, according to a previous setting.

[0153] Furthermore, if either the image height model or the mesh model's interpolation data is not extrapolated data, the interpolation data of either model is selected according to, for example, a previous setting.

[0154] 22 shows an example in which the imager 102 is shifted to the left. In the case of the interpolation position shown by a rectangle, the data is extrapolated in the mesh model, but is not extrapolated in the image height model. Therefore, in this case, the interpolation data of the image height model that is not extrapolated is selected.

[0155] Furthermore, in the case of the interpolation position indicated by a triangle, both the interpolation data of the image height model and the mesh model are extrapolated data. In this case, the interpolation data of the image height model is generated using the extrapolation data that is the third farthest from the region without extrapolation, and the interpolation data of the mesh model is generated using the extrapolation data that is the first farthest from the region without extrapolation. Therefore, in this case, the interpolation data of the mesh model that is generated using the extrapolation data that is the first farthest from the region without extrapolation is selected.

[0156] Based on the model selection determination result, for each interpolation position, either the interpolation data of the image height model or the interpolation data of the mesh model is selectively extracted (see FIG. 21(d)), and further smoothed by a low-pass filter (see FIG. 21(e)), thereby obtaining smoothed correction data (applied shading correction value data) of a predetermined resolution (see FIG. 21(f)).

[0157] The flowchart in Fig. 23 shows an example of a processing procedure corresponding to the generation processing of applied shading correction value data shown in Fig. 21. First, in step ST71, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the image height model correction data. Next, in step ST72, the camera control unit 131 generates interpolated data of a predetermined resolution that is interpolated into a mesh shape from the mesh model correction data.

[0158] Next, in step ST73, the camera control unit 131 determines whether to select the image height model or mesh model interpolation data for each interpolation position, based on the image height model correction data, the mesh model correction data, and further the imager shift information.

[0159] Next, in step ST74, the camera control unit 131 extracts, for each interpolation position, either the interpolation data of a predetermined resolution generated from the image height model correction data or the interpolation data of a predetermined resolution generated from the mesh model correction data based on the determination result, and further smoothes it to obtain mesh-like correction data of the predetermined resolution (applied shading correction value data).

[0160] In the generation process of the applied shading correction value data shown in FIG. 21, one of the interpolation data is selected for each interpolation position in accordance with the attributes of the interpolation data of the image height model and the mesh model at that interpolation position, thereby obtaining mesh-like correction data of a predetermined resolution (applied shading correction value data). For each interpolation position, the interpolation data of the image height model or the mesh model, whichever has higher accuracy, can be used, and more accurate data can be obtained as mesh-like correction data of a predetermined resolution.

[0161] "Processing for shading correction in imaging system" The flowchart in FIG. 24 shows an example of a procedure for processing related to shading correction in the imaging system 10 in FIG.

[0162] First, in step ST101, the camera control unit 131 determines whether or not both the image height model correction data and the mesh model correction data are present. When both the image height model correction data and the mesh model correction data are present, in step ST102, the camera control unit 131 generates interpolation data (applied shading correction value data) of a predetermined resolution that is interpolated in a mesh shape corresponding to the imager region (effective image region) from the image height model correction data and the mesh model correction data.

[0163] Next, in step ST103, the camera control unit 131 records the correction data (applied shading correction value data) having a predetermined resolution, which has been interpolated in a mesh shape, in metadata added by the RAW generation unit 138 in association with the RAW data file.

[0164] Next, in step ST104, the pre-processing unit 132 calculates the correction data (shading correction value) of the pixel of interest from the correction data (shading correction values) of the four points closest to the pixel of interest included in the correction data of a predetermined resolution (applied shading correction value data) generated by the camera control unit 131.

[0165] Next, in step ST105, the pre-processing unit 132 performs shading correction by multiplying the pixel of interest by the correction data (shading correction value) of the pixel of interest. Note that the processes in steps ST104 and ST105 are performed for each pixel in the imager area (effective image area).

[0166] Next, in step ST106, the camera control unit 131 records the correction data (applied shading correction value data) having a predetermined resolution that has been interpolated in a mesh shape in metadata that is added by the JPEG generation unit 138 in association with the JPEG file.

[0167] Furthermore, when neither the image height model correction data nor the mesh model correction data exists in step ST101, the camera control unit 131 determines whether or not the image height model correction data exists in step ST107. When the image height model correction data exists, the camera control unit 131 records the image height model correction data in metadata that is added in association with the RAW data file by the RAW generation unit 138 in step ST108.

[0168] Next, in step ST109, the pre-processing unit 132 calculates the correction data (shading correction value) of the pixel of interest from the correction data (provisional shading correction values) of two points that are closest to the pixel of interest among the correction data (provisional shading correction values) included in the image height model correction data.

[0169] Next, in step ST110, the pre-processing unit 132 performs shading correction by multiplying the pixel of interest by the correction data (shading correction value) of the pixel of interest. Note that the processes in steps ST109 and ST110 are performed for each pixel in the imager area (effective image area).

[0170] Next, in step ST111, the camera control unit 131 records the image height model correction data in metadata that is added in association with the JPEG file by the JPEG generation unit 138.

[0171] Furthermore, when the image height model correction data does not exist in step ST107, the camera control unit 131 determines whether or not the mesh model correction data exists in step ST112. When the mesh model correction data exists, the camera control unit 131 records the mesh model correction data in metadata that is added in association with the RAW data file by the RAW generation unit 138 in step ST113.

[0172] Next, in step ST114, the pre-processing unit 132 calculates the correction data (shading correction value) of the pixel of interest from the correction data (provisional shading correction values) of the four points closest to the pixel of interest among the correction data (provisional shading correction values) included in the mesh model correction data.

[0173] Next, in step ST115, the pre-processing unit 132 performs shading correction by multiplying the pixel of interest by the correction data (shading correction value) of the pixel of interest. Note that the processes in steps ST114 and ST115 are performed for each pixel in the imager area (effective image area).

[0174] Next, in step ST116, the camera control unit 131 records the image height model correction data in metadata that is added in association with the JPEG file by the JPEG generation unit 138.

[0175] Moreover, if no mesh model correction data exists in step ST112, no processing relating to shading correction is performed.

[0176] 24 shows an example in which correction data with a predetermined resolution interpolated in a mesh shape (applied shading correction value data), image height model correction data, and mesh model correction data are recorded in the metadata added in association with a RAW data file or a JPEG file. However, it is also possible to record data related to the determination of the applied shading correction value data generation method or the result of the determination in the metadata added in association with a RAW data file or a JPEG file.

[0177] Here, the data related to the determination of the applied shading correction value data generation method is, for example, correction model data, and the data related to the result of the determination of the applied shading correction value data generation method is predetermined area correction data for each of a plurality of areas of a predetermined size calculated from the correction model data, difference data, applied shading correction value data, data indicating shading correction values ​​for each of a plurality of pixels based on the applied shading correction value data, data indicating the applied shading correction value data generation method, etc.

[0178] 24 shows an example in which, when only image height model correction data exists or when only mesh model correction data exists, shading correction is performed using them as they are as applied shading correction value data. However, in these cases, it is also possible to generate mesh-shaped interpolation data with a predetermined resolution in the camera control unit 131 (see FIGS. 8(b) and (d)), and perform shading correction using the data as applied shading correction value data.

[0179] In this case, instead of recording image height model correction data or mesh model correction data, interpolation data (applied shading correction value data) of a predetermined resolution interpolated in a mesh shape is recorded in the metadata added in association with the RAW data file or the JPEG file. In this way, when only image height model correction data exists or when only mesh model correction data exists, by using the interpolation data of a predetermined resolution interpolated in a mesh shape as the applied shading correction value data, it is possible to obtain applied shading correction value data with a higher resolution (number of correction points).

[0180] As described above, in the imaging system 10 shown in FIG. 1, an applicable shading correction value data generation method for generating applicable shading correction value data to be applied to the captured image data is determined depending on which of a plurality of correction model data is present, each of which includes provisional shading correction values ​​for a plurality of areas within a frame of the captured image data, and the shading correction value data can be generated satisfactorily.

[0181] In addition, in the imaging system 10 shown in FIG. 1, applicable shading correction value data to be applied to the captured image data is generated based on a plurality of correction model data each including provisional shading correction values ​​for a plurality of areas within a frame of the captured image data, and the shading correction value data can be generated effectively.

[0182] <2. Second embodiment> [Example of imaging system configuration] Fig. 25 shows a configuration example of an imaging system 10A as a second embodiment. This second embodiment shows an example in which cloud processing is performed. In this Fig. 25, parts corresponding to those in Fig. 1 are given the same reference numerals, and detailed descriptions thereof will be omitted as appropriate.

[0183] The imaging system 10A includes a lens system (optical system) 101, an imager 102, an image signal processor (ISP) 103A, a lens driver 104, a vibration gyro 105, a flash memory 106, a public network 107 such as the Internet, and a cloud server 108.

[0184] The image signal processor 103A is configured as, for example, a system-on-a-chip (SoC). The image signal processor 103A includes a camera control unit 141, a compression unit 142, and a network interface 143.

[0185] The camera control unit 141 is configured by a microcomputer including a CPU, a ROM, and a RAM, similar to the camera control unit 131 in the image signal processor 103 of the imaging system 10 in Fig. 1. The camera control unit 141 controls the entire imaging system 10A by executing a program recorded in the ROM or the flash memory 106.

[0186] The compression unit 142 performs compression encoding processing on each of the RAW data and the shading correction data. The network interface 143 transmits the compressed and encoded data of the RAW data and the shading correction data to the cloud server 108 via the public network 107.

[0187] The cloud server 108 performs development processing including shading correction based on the RAW data and shading correction data sent from the image signal processor 103A via the public network 107, and generates image data in a format such as a JPEG file. This development processing is operated by, for example, a desktop application or a mobile application.

[0188] 26(a) shows an example of a detailed configuration of the image signal processor (ISP) 103A and the cloud server 108. The image signal processor (ISP) 103A has a correction data generation unit 144, compression units 142a and 142b, and network interfaces 143a and 143b.

[0189] The correction data generating unit 144 generates correction data (applied shading correction value data) with a predetermined resolution as shading correction data based on the image height model correction data and the mesh model correction data. Although detailed description is omitted, the correction data is generated by any one of the first to third methods described in the first embodiment.

[0190] The compression unit 142a performs compression encoding processing on the correction data (applied shading correction value data) with a predetermined resolution generated by the correction data generation unit 144. The network interface 143a transmits the compressed and encoded correction data (applied shading correction value data) with a predetermined resolution to the cloud server 108 via the public network 107.

[0191] The compression unit 142b performs a compression-encoding process on the RAW data. The network interface 143b sends the compressed and encoded RAW data to the cloud server 108 via the public network 107.

[0192] The cloud server 108 has decompression units 201a and 201b and a development processing unit 202. The decompression unit 201a performs decompression and decoding processing on the compression-encoded correction data (applied shading correction value data) with a predetermined resolution transmitted from the image signal processor (ISP) 103A via the public network 107, and transmits the compressed and encoded correction data to the development processing unit 202.

[0193] The decompression unit 201b performs decompression and decoding processing on the compressed and encoded RAW data sent from the image signal processor (ISP) 103A via the public network 107, and sends the data to the development processing unit 202. The development processing unit 202 performs development processing including shading correction using correction data (applied shading correction value data) of a predetermined resolution on the RAW data, and generates image data in a format such as a JPEG file.

[0194] 26(b) shows another example of a detailed configuration of the image signal processor (ISP) 103A and the cloud server 108. The image signal processor (ISP) 103A has compression units 142c, 142d, and 142b, and network interfaces 143c, 143d, and 143b.

[0195] The compression units 142c and 142d perform compression encoding processing on the image height model correction data and the mesh model correction data, respectively. The network interfaces 143c and 143d send the compressed and encoded image height model correction data and the mesh model correction data, respectively, to the cloud server 108 via the public network 107.

[0196] The compression unit 142b performs a compression-encoding process on the RAW data. The network interface 143b sends the compressed and encoded RAW data to the cloud server 108 via the public network 107.

[0197] The cloud server 108 has decompression units 201c, 201d, and 201b, a correction data generation unit 203, and a development processing unit 202. The decompression units 201c and 201d respectively perform decompression and decoding processing on the compressed and encoded image height model correction data and mesh model correction data transmitted from an image signal processor (ISP) 103A via a public network 107, and transmit the data to the correction data generation unit 203.

[0198] The correction data generation unit 203 generates correction data (applied shading correction value data) with a predetermined resolution as shading correction data based on the image height model correction data and the mesh model correction data, and sends the correction data to the development processing unit 202. Although detailed description will be omitted, the data is generated by any one of the first to third methods described in the first embodiment above.

[0199] The decompression unit 201b performs decompression and decoding processing on the compressed and encoded RAW data sent from the image signal processor (ISP) 103A via the public network 107, and sends the data to the development processing unit 202. The development processing unit 202 performs development processing including shading correction using correction data (applied shading correction value data) of a predetermined resolution on the RAW data, and generates image data in a format such as a JPEG file.

[0200] FIG. 27 shows yet another example of a detailed configuration of the image signal processor (ISP) 103A and the cloud server 108.

[0201] The image signal processor (ISP) 103A includes mesh-like interpolation data generating units 145c and 145d, a dividing unit 146, a smoothing unit 147, compression units 142c, 142e, and 142b, and network interfaces 143c, 143e, and 143b.

[0202] The mesh-like interpolation data generating unit 145c generates interpolation data with a predetermined resolution by interpolating the image height model correction data (see FIG. 8(a)) in a mesh shape (see FIG. 8(b)). The mesh-like interpolation data generating unit 145d generates interpolation data with a predetermined resolution by interpolating the mesh model correction data (see FIG. 8(c)) in a mesh shape (see FIG. 8(d)).

[0203] The division unit 146 performs division for each interpolation position between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data, thereby obtaining difference data of a predetermined resolution (see FIG. 8(e)). In this case, the interpolation data of the mesh model is divided by the interpolation data of the image height model. Then, the smoothing unit 147 smoothes the difference data of a predetermined resolution obtained by the division unit 146 using a low-pass filter, thereby obtaining smoothed difference data of a predetermined resolution (see FIG. 8(f)).

[0204] The compression unit 142c performs a compression encoding process on the image height model corrected data. The network interface 143c transmits the compressed and encoded image height model corrected data to the cloud server 108 via the public network 107.

[0205] The compression unit 142e performs a compression encoding process on the smoothed difference data obtained by the smoothing unit 147. The network interface 143e transmits the compression encoded smoothed difference data of a predetermined resolution to the cloud server 108 via the public network 107.

[0206] The compression unit 142b performs a compression-encoding process on the RAW data. The network interface 143b sends the compressed and encoded RAW data to the cloud server 108 via the public network 107.

[0207] The cloud server 108 has decompression units 201c, 201e, and 201b, a mesh-like interpolation data generation unit 204, a multiplication unit 205, and a development processing unit 202. The decompression unit 201c performs decompression and decoding processing on the compressed and encoded image height model correction data transmitted from an image signal processor (ISP) 103A via a public network 107, and transmits the compressed and encoded image height model correction data to the mesh-like interpolation data generation unit 204.

[0208] The mesh-like interpolation data generation unit 204 generates interpolation data of a predetermined resolution by interpolating the image height model correction data in a mesh shape, and sends the generated data to the multiplication unit 205. The expansion unit 201e performs expansion / decoding processing on the compression-encoded smoothed difference data of a predetermined resolution transmitted from the image signal processor (ISP) 103A via the public network 107, and sends the resulting data to the multiplication unit 205.

[0209] The multiplication unit 205 multiplies the interpolation data of a predetermined resolution (see FIG. 8(b)), which has been generated from the image height model correction data, by the smoothed difference data of a predetermined resolution (see FIG. 8(f)) for each interpolation position (see FIG. 8(g)) to obtain correction data of the predetermined resolution, that is, applied shading correction value data (see FIG. 8(h)), and sends the correction data to the development processing unit 202.

[0210] The decompression unit 201b performs decompression and decoding processing on the compressed and encoded RAW data sent from the image signal processor (ISP) 103A via the public network 107, and sends the data to the development processing unit 202. The development processing unit 202 performs development processing including shading correction using correction data (applied shading correction value data) of a predetermined resolution on the RAW data, and generates image data in a format such as a JPEG file.

[0211] The flowchart in FIG. 28 shows an example of a processing procedure of the image signal processor (ISP) 103A and the cloud server 108 corresponding to the configuration example in FIG.

[0212] First, in step ST81, the image signal processor 103A generates interpolation data of a predetermined resolution from the image height model correction data by interpolating the data in a mesh shape. Next, in step ST82, the image signal processor 103A generates interpolation data of a predetermined resolution from the mesh model correction data by interpolating the data in a mesh shape.

[0213] Next, in step ST83, the image signal processor 103A performs division for each interpolation position between the interpolation data of a predetermined resolution generated from the image height model correction data and the interpolation data of a predetermined resolution generated from the mesh model correction data to obtain difference data of a predetermined resolution. Next, in step ST84, the image signal processor 103A smoothes the difference data of the predetermined resolution to obtain smoothed difference data of the predetermined resolution.

[0214] Next, in step ST85, the image signal processor 103A compresses the image height model corrected data, the smoothed difference data with a predetermined resolution, and the RAW data, and transmits them to the cloud server 108.

[0215] Next, in step ST86, the cloud server 108 receives and decompresses the image height model corrected data, the smoothed difference data of a predetermined resolution, and the compressed data of the RAW data from the image signal processor 103 A. Next, in step ST87, the cloud server 108 generates interpolation data of a predetermined resolution that is interpolated in a mesh shape from the image height model corrected data.

[0216] Next, in step ST88, the cloud server 108 multiplies the interpolation data of a predetermined resolution generated from the image height model correction data by the smoothed difference data of a predetermined resolution for each interpolation position to obtain mesh-shaped correction data of a predetermined resolution (applied shading correction value data). Next, in step ST89, the cloud server 108 performs development processing including shading correction using the correction data of the predetermined resolution (applied shading correction value data) on the RAW data to generate image data in a format such as a JPEG file.

[0217] In the configuration example shown in FIG. 27, image height model correction data and smoothed difference data with a predetermined resolution are sent from an image signal processor (ISP) 103A to a cloud server 108 as shading correction data.

[0218] Both the image height model correction data and the mesh model correction data include correction data for shading design values, but by using differential data, the overlapping correction data for shading design values ​​is eliminated. In the configuration example shown in Fig. 27, the overlapping correction data for shading design values ​​is eliminated by using differential data, and the image height model correction data with a small data amount and the difference (residual) data with a small amplitude as shown in Fig. 29 are sent, so that the transmission data size can be reduced.

[0219] FIG. 29 shows an example of the correction amount of the image height model, the correction amount of the mesh model, and the correction amount of the difference (residual), and the amplitude of the correction amount of the difference (residual) is smaller than the amplitude of the correction amount of the image height model and the mesh model.

[0220] <3. Modifications> In the above embodiment, the image height model is used as the high resolution model and the mesh model is used as the low resolution model, but the selection of the model is not limited to this example. Fig. 30(a) shows a modified example of the model. In the figure, each model is described as having high resolution or low resolution, but this is merely an example, and which resolution it is is determined by comparing the models to be combined. Here, a case where the same models are combined as a combination of models is also considered. For example, a combination of coarse-fine mesh models with the same number of data points (for example, a combination of a coarse-fine mesh model A with 100 data points and a dense center and a coarse-fine mesh model B with 100 data points and a dense periphery) is considered.

[0221] Fig. 30(b) shows the details of each model. Note that, although the shading shape function model is not shown, it is a modification in which the correction data in the image height model is a table, but the correction amount is calculated as a function of the image height.

[0222] The polar coordinate model is a correction model that uses multiple image height tables by interpolating according to the angle θ from the central position of the optical axis. For example, it refers to a model that defines two image height tables in the angle 0 degree direction and 180 degree direction. The resolution can be high or low depending on the number of divisions in the θ direction. The coarse-dense function model is a correction model that increases the correction resolution at the edges of the screen (the resolution is coarse in the center of the screen) because shading often changes more rapidly towards the edges of the screen.

[0223] The screen edge function model is an extreme version of the coarse-fine function model. In this case, no correction value is specified for the center of the screen, and correction values ​​are specified only for the edges of the screen. By eliminating the correction value for the center of the screen, the correction resolution for the edges of the screen is finer. The coarse mesh model is a correction model used when there is a communication capacity limit in communication of the correction table and the capacity needs to be reduced. This mesh model is a correction model with a relatively coarse resolution of the mesh model.

[0224] In addition, although the preferred embodiment of the present disclosure has been described in detail with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0225] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0226] The present technology can also be configured as follows. (1) an applied shading correction value data generation method determination unit is provided which, depending on which correction model data is present among a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of captured image data, determines an applied shading correction value data generation method for generating applied shading correction value data including shading correction values ​​for each of a plurality of areas having a predetermined size smaller than the plurality of areas in the correction model data, to be applied to the captured image data; Information processing device. (2) the applied shading correction value data generation method determination unit determining a method for generating the applied shading correction value data according to the number of correction model data included in the plurality of correction model data; The information processing device according to (1). (3) the applied shading correction value data generation method determination unit In the case where only one correction model data is included among the plurality of correction model data, determining the applied shading correction value data generation method to be a single correction model data utilization generation method based on only the one correction model data among the plurality of correction model data; The information processing device according to (2). (4) The single correction model data utilization generation method includes: a method for setting data of shading correction values ​​for each of a plurality of areas of the predetermined size based on the one correction model data as the applied shading correction value data, The information processing device according to (3). (5) the applied shading correction value data generation method determination unit When the plurality of correction model data includes first correction model data and second correction model data, The applied shading correction value data generation method is a multiple correction model data generation method based on both the first correction model data and the second correction model data. The information processing device according to any one of (2) to (4). (6) An applied shading correction value data generating unit that generates the applied shading correction value data based on the applied shading correction value data generating method. The information processing device according to any one of (1) to (5). (7) An association unit that associates the applied shading correction value data generated by the applied shading correction value data generation unit with the captured image data. The information processing device according to (6). (8) An associating unit that associates data related to the determination of the applied shading correction value data generation method or a result of the determination with the captured image data. The information processing device according to any one of (1) to (7). (9) An image processing unit that performs shading correction of the captured image data by using a shading correction value for each of a plurality of pixels based on the applied shading correction value data generated by the applied shading correction value data generation unit. The information processing device according to any one of (1) to (8). (10) The image processing device further includes an associating unit that associates the applied shading correction value data generated by the applied shading correction value data generating unit with corrected captured image data generated by the image processing unit performing shading correction on the captured image data. The information processing device according to (9). (11) A method for generating applied shading correction value data, the method being for generating applied shading correction value data including shading correction values ​​for a plurality of areas of a predetermined size smaller than the plurality of areas in the correction model data, to be applied to the captured image data, according to which correction model data is included among a plurality of correction model data including provisional shading correction values ​​for a plurality of areas in a frame of the captured image data. Information processing methods. (12) An applied shading correction value data generating unit generates applied shading correction value data including shading correction values ​​for a plurality of areas of a predetermined size smaller than the plurality of areas in the correction model data, based on a plurality of correction model data each including a provisional shading correction value for each of a plurality of areas in a frame of the captured image data, to be applied to the captured image data. Information processing device. (13) The plurality of correction model data includes actual measurement value correction model data based on actual measurement values. The information processing device according to (12). (14) The actual measurement value correction model data is mesh model data. The information processing device according to (13). (15) The plurality of correction model data includes design value model correction data based on design values, The information processing device according to any one of (12) to (14). (16) The design value model correction data is image height model data based on an image height distance. The information processing device according to (15). (17) The plurality of correction model data include first correction model data and second correction model data, The first correction model data has a higher degree of freedom in setting the shading correction value for each area than the second correction model data. The information processing device according to any one of (12) to (16). (18) The plurality of correction model data include first correction model data and second correction model data, the first correction model data has a lower correction resolution in the area than the second correction model data; The information processing device according to any one of (12) to (17). (19) The plurality of correction model data include first correction model data and second correction model data, The applied shading correction value data generating unit generating the applied shading correction value data based on correction value difference data for each predetermined area, the correction value difference data being data of difference values ​​of shading correction values ​​for each of a plurality of areas of the predetermined size generated based on the first correction model data and the second correction model data; The information processing device according to any one of (12) to (18). (20) the second correction model data has a higher correction resolution for the area than the first correction model data; The applied shading correction value data generating unit generating the applied shading correction value data based on predetermined area correction value data, which is data of shading correction values ​​for each of a plurality of areas of the predetermined size based on the second correction model data, and the predetermined area correction value difference data; The information processing device according to (19). (21) The plurality of correction model data include first correction model data and second correction model data, The applied shading correction value data generating unit generating the applied shading correction value data based on a magnitude of a difference value of shading correction values ​​for each of the predetermined areas between first correction value data for each of the predetermined areas, which is data of shading correction values ​​for each of the plurality of areas of the predetermined size based on the first correction model data, and second correction value data for each of the predetermined areas, which is data of shading correction values ​​for each of the plurality of areas of the predetermined size based on the second correction model data; The information processing device according to any one of (12) to (18). (22) The applied shading correction value data generating unit generating the applied shading correction value data in response to imager shift information relating to a shift of an imager; The information processing device according to any one of (12) to (18). (23) An extrapolation unit that extrapolates the provisional shading correction value to one or more areas where the provisional shading correction value is missing in the plurality of correction model data, The applied shading correction value data generating unit generating the applied shading correction value data based on the number of pieces of provisional shading correction value data extrapolated to each of the plurality of correction model data by the extrapolation unit in accordance with the imager shift information; The information processing device according to (22). (24) An image processing unit that performs shading correction on the captured image data by using the applied shading correction value data generated by the applied shading correction value data generation unit. The information processing device according to any one of (12) to (23). [Explanation of symbols]

[0227] 10,10A... Imaging system 101...Lens system 101a Image height table for shading correction 102 Imager 103, 103A Image signal processor 104 Lens driver 105...Vibration gyro 106 Flash memory 106a···Mesh table for shading correction 107 Public Network 108···Cloud Server 131 Camera control unit 132 Pre-processing section 133 Synchronization section 134...YC generation section 135 Post-processing section 136...JPEG generation section 137···Output Interface 138...RAW generation section 139 Output Interface 141 Camera control unit 142, 142a to 142e...Compression section 143, 143a to 143e Network Interface 144...Correction data generation unit 145c, 145d: Mesh-shaped interpolation data generation unit 146...Division section 147 Smooth section 201a~201e...Extension part 202 Development processing section 203... Correction data generation unit 204 Mesh-shaped interpolation data generation unit 205... Multiplication unit

Claims

1. an applied shading correction value data generation method determination unit that determines an applied shading correction value data generation method for generating applied shading correction value data including shading correction values ​​for second areas of a predetermined size smaller than the first areas in the correction model data, to be applied to the captured image data, according to which correction model data is included among a plurality of correction model data each including a provisional shading correction value for each of a first plurality of areas in a frame of the captured image data; Information processing device.

2. The applied shading correction value data generation method determination unit determining a method for generating the applied shading correction value data according to the number of correction model data included in the plurality of correction model data; The information processing device according to claim 1 .

3. The applied shading correction value data generation method determination unit In the case where only one correction model data is included among the plurality of correction model data, determining the applied shading correction value data generation method to be a single correction model data utilization generation method based on only the one correction model data among the plurality of correction model data; The information processing device according to claim 2 .

4. The single correction model data utilizing generation method includes: a method for setting data of shading correction values ​​for each of the second areas of the predetermined size based on the one correction model data as the applied shading correction value data, The information processing device according to claim 3 .

5. The applied shading correction value data generation method determination unit In the case where the plurality of correction model data includes first correction model data and second correction model data, The applied shading correction value data generation method is a multiple correction model data generation method based on both the first correction model data and the second correction model data. The information processing device according to claim 2 .

6. an applied shading correction value data generating unit that generates the applied shading correction value data based on the applied shading correction value data generating method; The information processing device according to claim 1 .

7. an association unit that associates the applied shading correction value data generated by the applied shading correction value data generation unit with the captured image data; The information processing device according to claim 6.

8. an associating unit that associates data related to the determination of the applied shading correction value data generation method or a result of the determination with the captured image data; The information processing device according to claim 1 .

9. an image processing unit that performs shading correction on the captured image data by using shading correction values ​​for each of a plurality of pixels based on the applied shading correction value data generated by the applied shading correction value data generation unit; The information processing device according to claim 6.

10. an associating unit that associates the applied shading correction value data generated by the applied shading correction value data generating unit with corrected captured image data generated by the image processing unit by performing shading correction on the captured image data; The information processing device according to claim 9.

11. a step of determining an applied shading correction value data generating method for generating applied shading correction value data including shading correction values ​​for second areas of a predetermined size smaller than the first areas in the correction model data, to be applied to the captured image data, according to which correction model data is included among a plurality of correction model data each including a provisional shading correction value for each of a first plurality of areas in a frame of the captured image data; Information processing methods.

12. an applied shading correction value data generating unit that generates applied shading correction value data including second shading correction values ​​for a plurality of areas having a predetermined size smaller than the first areas in the correction model data, based on a plurality of correction model data each including a provisional shading correction value for each of a first plurality of areas in a frame of the captured image data, to be applied to the captured image data; Information processing device.

13. the plurality of correction model data include first correction model data and second correction model data; the first correction model data has a higher degree of freedom in setting the shading correction value for each area than the second correction model data; The information processing device according to claim 12.

14. the plurality of correction model data include first correction model data and second correction model data; the first correction model data has a lower correction resolution in the area than the second correction model data; The information processing device according to claim 12.

15. the plurality of correction model data include first correction model data and second correction model data; The applied shading correction value data generating unit generating the applied shading correction value data based on correction value difference data for each predetermined area, the correction value difference data being data of difference values ​​of shading correction values ​​for each of the second plurality of areas of the predetermined size generated based on the first correction model data and the second correction model data; The information processing device according to claim 12.

16. the second correction model data has a higher correction resolution for the area than the first correction model data; The applied shading correction value data generating unit generating the applied shading correction value data based on predetermined area correction value data, which is data of shading correction values ​​for each of the second plurality of areas of the predetermined size based on the second correction model data, and the predetermined area correction value difference data; The information processing device according to claim 15.

17. the plurality of correction model data include first correction model data and second correction model data; The applied shading correction value data generating unit generating the applied shading correction value data based on a magnitude of a difference value of shading correction values ​​for each of the predetermined areas between first correction value data for each of the predetermined areas, which is data of shading correction values ​​for each of the second plurality of areas of the predetermined size based on the first correction model data, and second correction value data for each of the predetermined areas, which is data of shading correction values ​​for each of the second plurality of areas of the predetermined size based on the second correction model data; The information processing device according to claim 12.

18. The applied shading correction value data generating unit generating the applied shading correction value data in response to imager shift information relating to a shift of an imager; The information processing device according to claim 12 .

19. an extrapolation unit that extrapolates the provisional shading correction value to one or more areas where the provisional shading correction value is missing in the plurality of correction model data; The applied shading correction value data generating unit generating the applied shading correction value data based on the number of pieces of provisional shading correction value data extrapolated to each of the plurality of correction model data by the extrapolation unit in accordance with the imager shift information; The information processing device according to claim 18.

20. an image processing unit that performs shading correction on the captured image data by using the applied shading correction value data generated by the applied shading correction value data generation unit; The information processing device according to claim 12.

Images