Image processing device, imaging device, and image processing method

Abstract

This image processing device for developing YC-RAW data comprises an image processing unit that: subjects YC-RAW data, which is image data obtained by performing a pre-white balance processing, a de-mosaic processing and a YC conversion processing with respect to image data of image sensor output format, to a processing of conversion to an image sensor output format; and further subjects the resultant YC-RAW data to a reverse-pre-white balance processing, which is associated with the pre-white balance processing, and then to a re-mosaic processing for getting the resultant data back to the state as before the de-mosaic processing.

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an imaging device, and an image processing method, and in particular to a technology for developing RAW images. [Background technology]

[0002] When recording an image, not only image data generated by developing the output of an image sensor, such as JPEG (Joint Photographic Experts Group) image data, but also RAW data is recorded. RAW data is recorded in a format that reproduces the signal obtained from the image sensor almost exactly as it is, so that the recorded RAW data can be post-processed by specifying desired development parameters and performing development processing to adjust gradation, color, etc. This allows users to obtain developed images that are more to their liking than methods that perform color reproduction processing, etc. within the imaging device.

[0003] RAW data may refer to image data in the format output by an image sensor, and one method of recording RAW data is to record the image data in the format output by an image sensor, for example, the R (red), G (green), and B (blue) signals generated by a primary color filter array, as is. For the sake of explanation, an example of this type of RAW data will be called RGB-RAW data.

[0004] In order to reduce the amount of data when saved, some signal processing, including white balance processing, is performed within the imaging device, separating the image into luminance and color difference signals, and recording them as RAW data in YCbCr format. RAW data that has undergone this type of YC conversion processing is called YC-RAW data.

[0005] The following Patent Document 1 discloses processing of an image equivalent to YC-RAW data. Furthermore, Patent Document 2 below discloses processing related to the reliability of the white balance of RAW data. In the following, "white balance" will also be abbreviated as "WB." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-5245 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-176272 Summary of the Invention [Problem to be solved by the invention]

[0007] Incidentally, to record an image in YCbCr format as YC-RAW data, it is necessary to first perform WB processing on the image sensor output, for example, a Bayer-array RGB signal, and then perform demosaic processing after WB processing to estimate the missing color value of each pixel in the image, thereby creating a color image. Because these missing color values ​​are estimated using color information from surrounding pixels, if the WB processing value (WB gain) is not set correctly, the missing color value of each pixel in the image cannot be estimated correctly. As can be seen from Patent Document 2, WB processing is affected by various factors, such as the color of the subject, the light source, and the detection method, making it difficult to accurately calculate in a short time. For example, it is difficult to quickly set an appropriate WB value during image capture. Furthermore, demosaicing may worsen color shifts. As a result, when the YC-RAW data is subsequently developed, a developed image, such as JPEG image data, that still retains the effects of color shift is generated.

[0008] Therefore, this disclosure proposes image processing that corrects the effects of color shift, which can be applied when developing YC-RAW data, for example. [Means for solving the problem]

[0009] The image processing device according to the present technology is equipped with an image processing unit that performs conversion processing into the image sensor output format on YC-RAW data, which is image data that has been subjected to pre-white balance processing, demosaic processing, and YC conversion processing on image data in the image sensor output format, and further performs inverse pre-white balance processing on the pre-white balance processing, and re-mosaic processing to return the image to the state before the demosaic processing. For example, simple pre-white balance processing or demosaic processing performed when capturing YC-RAW data can cause color shifts when the YC-RAW data is developed compared to the original image. The above configuration corrects color shifts in the YC-RAW data.

[0010] The image processing device according to the present technology also includes a recording control unit that performs a conversion process to the image sensor output format on YC-RAW data, which is image data obtained by performing pre-white balance processing, demosaic processing, and YC conversion processing on image data in the image sensor output format, and further performs an inverse pre-white balance process on the pre-white balance process and a re-mosaic process that returns the image data to the state before the demosaic process, or records the YC-RAW data obtained based on the image data in the image sensor output format obtained after the re-mosaic process on a recording medium. That is, the RAW data, which is image data after color shift correction processing including inverse pre-white balance processing and re-mosaic processing, is recorded by overwriting the original YC-RAW data, or is recorded together with the original YC-RAW data.

[0011] An imaging device according to the present technology includes an imaging unit that obtains captured image data using an image sensor; an image processing unit that performs pre-white balance processing, demosaic processing, and YC conversion processing on the image data in an image sensor output format obtained by the imaging unit to generate YC-RAW data; and a control unit that performs processing to associate sensor array information that represents the pixel array of the image sensor with the YC-RAW data as metadata used in re-mosaic processing that returns the image to a state before the demosaic processing, and pre-white balance information including parameters for the pre-white balance processing as metadata used in inverse pre-white balance processing of the pre-white balance processing. For example, sensor array information and pre-white balance information can be added to the metadata associated with YC-RAW data so that they can be referenced during development processing. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram of an imaging device according to an embodiment of the present technology; [Figure 2] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 3] 10A to 10C are explanatory diagrams of RAW imaging processing according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of an example of a pixel array of an image sensor. [Figure 5] FIG. 10 is an explanatory diagram of RGB plane signals before re-mosaicing. [Figure 6] 10 is a flowchart illustrating a process for associating metadata with YC-RAW data according to an embodiment. [Figure 7] 10 is a flowchart illustrating a process for associating metadata with YC-RAW data according to an embodiment. [Figure 8] FIG. 10 is an explanatory diagram of a RAW development process without color shift correction. [Figure 9] FIG. 10 is an explanatory diagram of color shift during RAW development. [Figure 10] FIG. 2 is an explanatory diagram of RAW development processing according to the first embodiment. [Figure 11] FIG. 10 is an explanatory diagram of a signal after re-mosaicing. [Figure 12] FIG. 10 is an explanatory diagram of RAW development processing according to the second embodiment. [Figure 13] FIG. 10 is an explanatory diagram of an example of an image generated in the second embodiment. [Figure 14] FIG. 10 is an explanatory diagram of RAW development processing according to the third embodiment. [Figure 15] 13 is a flowchart of a determination process according to the fourth embodiment. [Figure 16] FIG. 10 is an explanatory diagram of a user setting screen according to the embodiment. [Figure 17] 10A and 10B are explanatory diagrams illustrating examples of warning displays according to an embodiment. [Figure 18] 10A and 10B are explanatory diagrams illustrating an example of displaying a color misalignment location according to an embodiment. [Figure 19] 10A to 10C are explanatory diagrams of examples of presentation before and after color misregistration correction according to an embodiment. [Figure 20] FIG. 10 is an explanatory diagram of an image division process according to an embodiment. [Figure 21] 13 is a flowchart of another determination process according to the fourth embodiment. [Figure 22] 13 is a flowchart of still another determination process according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] The embodiments will be described below in the following order. <1. Configuration of imaging device> <2. Configuration of information processing device> <3. YC-RAW imaging processing> <4. Comparative example of YC-RAW development> <5. First embodiment: YC-RAW development> <6. Second embodiment: YC-RAW development> 7. Third Embodiment: Overwriting of YC-RAW 8. Fourth Embodiment: Processing with Determination of Color Misalignment Correction Execution <9. Summary and Variations>

[0014] <1. Configuration of imaging device> It is assumed that the image processing device according to the embodiment of the present technology is installed as an image processing unit in an imaging device (camera) or an information processing device that performs RAW development, etc. Furthermore, the imaging device or information processing device itself that is equipped with these image processing units can also be considered as the image processing device according to the embodiment of the present technology.

[0015] First, an example of the configuration of an imaging device 1 will be described with reference to FIG. This imaging device 1 is equipped with an image processing unit 20 that generates YC-RAW data according to the image capture and develops the YC-RAW data, and this image processing unit 20, or the imaging device 1 equipped with the image processing unit 20, can be considered an example of an image processing device of the present technology.

[0016] The YC-RAW data generated by the imaging device 1 is image data obtained by performing pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format obtained by imaging, such as RGB format. Resize processing may also be performed to change the pixel size (number of pixels). However, this YC-RAW data has not undergone any image creation processing such as color reproduction processing, gamma processing, or sharpness processing, and can be considered a type of RAW data.

[0017] The imaging device 1 has, for example, a lens system 11, an imaging unit 12, a recording control unit 14, a display unit 15, a communication unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, an image processing unit 20, a buffer memory 21, a driver unit 22, and a sensor unit 23.

[0018] The lens system 11 includes lenses such as a zoom lens and a single focus lens, an aperture mechanism, etc. The lens system 11 guides light (incident light) from a subject and collects the light on the imaging unit 12.

[0019] The imaging unit 12 includes an image sensor 12a, such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The imaging unit 12 performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectrically converting the light received by the image sensor 12a, and then performs A / D (Analog / Digital) conversion on the electrical signal, and outputs the imaging signal as digital data to the image processing unit 20 and the camera control unit 18 in the subsequent stages.

[0020] The image processing unit 20 is configured as an image processor, for example, using a DSP (Digital Signal Processor). The image processing unit 20 performs various types of signal processing on the digital signal (captured image signal) from the imaging unit 12. The various types of processing include, for example, preprocessing, white balance (WB) processing, demosaic processing, color / gradation processing, YC conversion processing, resolution conversion processing, and various detection processes.

[0021] As pre-processing, the image processing unit 20 performs defect correction and black level adjustment on the signal from the image sensor 12a, for example. As for WB processing, the image processing unit 20 performs simple WB processing for generating YC-RAW data, or performs WB processing by determining a high-precision WB value, for example, by full pixel detection, or performs original WB processing using a WB value specified by the user, as will be described later. In this disclosure, for the sake of clarity, the simple WB processing performed during image capture to generate YC-RAW data is referred to as "pre-WB processing" ("pre-white balance processing"). Furthermore, WB processing performed by obtaining a highly accurate WB value using full pixel detection or the like, or WB processing performed with a user-specified WB value, i.e., WB processing that is reflected in the final developed image, is referred to as "main WB processing" ("main white balance processing"). Furthermore, the image processing unit 20 may also perform inverse pre-WB processing, which cancels the pre-WB processing by using a WB value that is the inverse of that used in the pre-WB processing.

[0022] The image processing unit 20 performs demosaic processing so that the image data for each pixel has all the color components R, G, and B. The image processing unit 20 may also perform re-mosaic processing to return the demosaic-processed image data to the state before the demosaic processing. In the YC conversion process, the image processing unit 20 generates (separates) a luminance signal and a color difference signal from image data in RGB format, and performs processing to convert the image data into image data in YCbCr format. Color and gradation processing involves adjusting gradation, saturation, tone, contrast, etc., in order to create an image.

[0023] The image processing unit 20 performs, for example, the above-mentioned processes as needed, and generates YC-RAW data and developed image data from the image data from the image sensor 12a. In this case, resolution conversion and file creation processing may be performed. In the file creation processing, image data is subjected to, for example, compression encoding for recording or communication, formatting, and generation and addition of metadata to create files for recording or communication. For example, still image files can be generated in formats such as JPEG, TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF (High Efficiency Image File Format), YUV422, YUV420, etc. It is also possible to generate image files in formats such as MP4, which is used for recording MPEG-4 compliant video and audio.

[0024] The buffer memory 21 is formed of, for example, a DRAM (Dynamic Random Access Memory), and is used for temporarily storing image data in the image processing unit 20 during the various processes described above.

[0025] The recording control unit 14 performs recording and reproduction on a recording medium such as a nonvolatile memory, and performs processing to record image files such as moving image data and still image data on the recording medium. The recording control unit 14 may take a variety of actual forms. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and its write / read circuit. The recording control unit 14 may also take the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0026] The display unit 15 is a display unit that displays various information to the user, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display that is arranged on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18. For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14.

[0027] Furthermore, image data of the captured image that has been resolution-converted for display by the image processing unit 20 is supplied to the display unit 15, and the display unit 15 may display based on the image data of the captured image in response to an instruction from the camera control unit 18. This allows the display of a so-called through image (live view image of the subject), which is an image captured while checking the composition or recording a video. Furthermore, based on instructions from the camera control unit 18, the display unit 15 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), on the screen.

[0028] The communication unit 16 performs wired or wireless data communication and network communication with external devices, such as transmitting and outputting still image files and video files containing captured image data and metadata to external information processing devices, display devices, recording devices, playback devices, etc. Furthermore, the communication unit 16 serves as a network communication unit, and performs communication via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various types of data to and from servers, terminals, etc. on the network. Servers on the network include so-called cloud servers, and the imaging device 1 can transmit and receive various types of information to and from the cloud servers via the communication unit 16. Furthermore, the imaging device 1 may be configured to be able to communicate information with, for example, a PC, a smartphone, a tablet terminal, etc. via the communication unit 16 by, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark) communication, near-field communication such as NFC (Near Field Communication), infrared communication, etc. Furthermore, the imaging device 1 and other devices may be able to communicate with each other via wired connection communication.

[0029] Therefore, the imaging device 1 can transmit image data and metadata via the communication unit 16 to an information processing device 70, which will be described later.

[0030] The operation unit 17 collectively refers to input devices that allow the user to input various operations. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. The operation unit 17 detects the user's operation, and a signal corresponding to the input operation is sent to the camera control unit 18 .

[0031] The camera control unit 18 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 comprehensively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 19 may be a memory area built into the microcomputer chip that serves as the camera control unit 18, or may be configured as a separate memory chip.

[0032] The camera control unit 18 controls the entire imaging device 1 by executing a program stored in the ROM or flash memory of the memory unit 19 . For example, the camera control unit 18 controls the shutter speed of the imaging unit 12, instructs various signal processing in the image processing unit 20, controls the imaging operation and recording operation in response to user operations, controls the playback operation of recorded image files, controls the operation of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, and controls the user interface operation, among other operations.

[0033] The RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 18 to process various data, and is used to temporarily store data, programs, and the like. The ROM and flash memory (non-volatile memory) in memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, various setting information, etc.

[0034] The driver section 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, and cause the focus lens and zoom lens to move, and the aperture blades of the aperture mechanism to open and close.

[0035] The sensor unit 23 collectively represents various sensors mounted on the imaging device. When an IMU (inertial measurement unit), for example, is installed as the sensor unit 23, angular velocity can be detected using a three-axis angular velocity (gyro) sensor of pitch, yaw, and roll, and acceleration can be detected using an acceleration sensor. The sensor unit 23 may also include, for example, a position information sensor, an illuminance sensor, a distance measurement sensor, and the like.

[0036] Various information detected by the sensor unit 23, such as position information, distance information, illuminance information, IMU data, etc., is added to the captured image as metadata together with date and time information managed by the camera control unit 18.

[0037] By providing an illuminance sensor capable of detecting color temperature as the sensor unit 23, the camera control unit 18 can estimate the light source used during image capture and, for example, can add light source information to metadata. Of course, it is also possible to use an illuminance sensor separate from the image capture device 1.

[0038] <2. Configuration of information processing device> Next, an example of the configuration of the information processing device 70 will be described with reference to FIG. The information processing device 70 is a device such as a computer that is capable of information processing, particularly image processing. Specific examples of the information processing device 70 include personal computers (PCs), mobile terminal devices such as smartphones and tablets, mobile phones, video editing devices, and video playback devices. The information processing device 70 may also be a computer configured as a server device or a computing device in cloud computing.

[0039] This information processing device 70 is equipped with an image processing unit 20 that performs development processing on YC-RAW image data, and this image processing unit 20, or an information processing device 70 equipped with the image processing unit 20, can be considered an example of an image processing device of the present disclosure.

[0040] The CPU 71 of the information processing device 70 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a recording medium into the RAM 73 by a recording control unit 79. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate.

[0041] The image processing unit 20 performs, for example, white balance processing, demosaic processing, color and gradation processing, YC conversion processing, resolution conversion processing, various detection processing, etc. on the image data. In particular, these processes are used to perform development processing on the YC-RAW data, generating a developed image, for example, JPEG image data. In the case of the image processing unit 20 in this information processing device 70, the YC-RAW data to be developed is, for example, image data captured by the imaging device 1 and imported into, for example, a recording medium in the recording control unit 79 via communication or a recording medium. In this case, metadata generated by the imaging device 1 is also associated with the YC-RAW data.

[0042] The function of the image processing unit 20 to perform development processing of YC-RAW data may be provided as a function within the CPU 71. The image processing unit 20 may also be realized by a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, or the like.

[0043] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 20 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0044] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0045] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD or organic EL panel or the like, and an audio output unit 78 made up of a speaker or the like. The display unit 77 is configured by, for example, a display device provided in the housing of the information processing device 70, or a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., i.e., GUI (Graphical User Interface), based on instructions from the CPU 71.

[0046] The input / output interface 75 may also be connected to a recording control unit 79 and a communication unit 80 . The recording control unit 79 can record data to be processed and various programs on a recording medium such as a disk in a hard disk drive (HDD) or a solid-state memory. When the information processing device 70 functions as the image processing device of the present disclosure, the recording control unit 79 is expected to record the YC-RAW data and metadata to be processed on a recording medium, and to record image data generated by the development process, such as JPEG image data, on a recording medium. The recording control unit 79 can also record and read out programs for development processing from and to a recording medium.

[0047] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like. Communication with the imaging device 1, particularly reception of image data and the like, is performed by a communication unit 80.

[0048] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. Drive 81 allows data files such as image files and various computer programs to be read from removable recording medium 82. The read data files are stored in recording control unit 79, and images and sounds contained in the data files are output on display unit 77 and audio output unit 78. Furthermore, computer programs and the like read from removable recording medium 82 are recorded on a recording medium in recording control unit 79 as necessary.

[0049] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72 or a recording medium in the recording control unit 79.

[0050] <3. YC-RAW imaging processing> The following describes the YC-RAW data imaging process performed by the above-described imaging device 1. That is, this process involves capturing an image, generating YC-RAW data as RAW data before development, and recording it on a recording medium or transmitting it to an external device.

[0051] 3 shows the flow of processing in the image processing unit 20 when performing RAW imaging processing. The diagram shows the camera control unit 18 and bus 101, which indicates that each process in the image processing unit 20 is performed in accordance with control signals and parameter settings from the camera control unit 18. Each process performed in the image processing unit 20 is shown with a step number (S100 to S104), which indicates a processing procedure up to the generation of YC-RAW data, and is part of the processing performed in the image processing unit 20.

[0052] As described above, image data ISDT from the image sensor 12a is supplied to the image processing unit 20. This image data ISDT is image data in an image sensor output format, such as RGB format RAW image data (RGB-RAW data) with R values, G values, and B values ​​remaining in the pixel arrangement of the image sensor 12a.

[0053] As for the imaging process when recording YC-RAW data, the image processing unit 20 performs signal processing on the image data ISDT digitized by the imaging unit 12 in the order of pre-WB processing in step S100, demosaic processing in step S101, and resizing processing in step S102. Then, in step S103, YC conversion processing is performed to convert the signal into luminance and color difference signals (YCbCr signals), resulting in YC-RAW data, which is output in interface processing in step S104. This output YC-RAW data may be recorded on a recording medium by the recording control unit 14, or may be transmitted and output by the communication unit 16, for example.

[0054] Each process will be explained. The pre-WB processing in step S100 estimates a WB value (WB gain) by simple detection of the image data ISDT generated by the imaging unit 12, and adjusts the RGB signal level by applying the WB gain to, for example, an RGB signal.

[0055] The WB used here is the WB value for internal processing, and it is desirable to use a value that makes achromatic subjects as achromatic as possible. However, in the case of high-speed processing such as recording RAW data, in order to shorten the calculation time for determining pre-WB, WB processing is performed using the previous frame value in live view, or WB processing using simple detection or low-precision high-speed calculation. If the WB value of the previous frame is used, a difference may occur between the appropriate value for the current frame and the WB value of the previous frame. Furthermore, if simple detection or low-precision high-speed calculation is used, it is difficult to obtain an appropriate WB value for the current frame, and a difference may occur between the WB value that results in a completely achromatic color.

[0056] In the demosaic processing in step S101, demosaic processing is performed using the image data processed in the pre-WB processing, and a color image is created by estimating the missing color value of each pixel in the image. These missing color values ​​are estimated using color information from surrounding pixels. There are several common demosaicing methods, including interpolation-based methods, feature-based methods, and Bayesian methods, but in general, there is a tendency to lighten the color of the interpolated signal in order to suppress color artifacts. As a result, color misalignment worsens after the RGB signal, which has been affected by the pre-WB difference, is demosaiced.

[0057] Note that the demosaic processing described here often generates an image signal in which one pixel represents each color component, for example, an image signal of the three primary colors, from an image signal in which one pixel represents one color component, but it is not limited to the three primary colors and may also generate a four-color image, etc. Furthermore, although the pixel arrangement of the image sensor 12a is typically a Bayer pattern, the technology of the present disclosure can also be applied to other image arrangements. In other words, various specifications are conceivable for the image sensor 12a.

[0058] For example, FIG. 4 shows examples of pixel arrangements of the image sensor 12a when a primary color filter is used, when a complementary color filter is used, and when a W-RGB filter is used.

[0059] The example of the primary color filter shown in the figure is an example of a Bayer array in which G, B, and R are included in a ratio of 2:1:1, and the array is based on blocks of "R", "G", "G", and "B" pixels. The first pixel is an R pixel. In the case of a complementary color filter, the basic arrangement is a block of four complementary color pixels: "Cy (cyan)," "Ye (yellow)," "G," and "Mg (magenta)." The first pixel is a Cy pixel. In the case of a W-RGB filter, the basic arrangement is a block of four pixels: "B," "W (white)," "R," and "G." The first pixel is a B pixel. Of course, these are just examples of pixel arrangements of the image sensor 12a, and various other pixel arrangements are conceivable.

[0060] If the image sensor 12a has a pixel array using primary color filters, for example, image data called RGB planes, as shown in FIG. 5, will be obtained by demosaicing.

[0061] The resizing process in step S102 in Fig. 3 reduces the amount of data by converting the image size in order to save RAW data quickly and save memory. Here, it is preferable to use interpolation processing rather than simple downsampling to avoid image quality degradation due to resizing. Note that there are cases where resizing processing is not performed.

[0062] The YC conversion process in step S103 converts the demosaic processed image data into luminance data and color difference data, thereby generating RAW image data as YC-RAW data. This YC-RAW data is recorded within the imaging device 1, or transmitted to an external device such as the information processing device 70 and recorded therein.

[0063] Up to this point, we have explained the imaging and recording process of YC-RAW data, but at this time, the processing is instructed and metadata is associated by the camera control unit 18. An example of the processing performed by the camera control unit 18 for this purpose is shown in FIG.

[0064] 6, in step S1, the camera control unit 18 instructs the image processing unit 20 to generate YC-RAW data. That is, the camera control unit 18 instructs the image processing unit 20 to generate and record YC-RAW data from the image data ISDT captured by the imaging unit 12. This causes the image processing unit 20 to execute the processes in FIG. 3 described above.

[0065] In step S2, the camera control unit 18 instructs the camera control unit 18 on the parameters of the pre-WB processing, that is, the WB value. For example, the camera control unit 18 calculates the WB value based on simple detection processing by the image processing unit 20, and instructs the calculated WB value to the image processing unit 20. Alternatively, the camera control unit 18 instructs the image processing unit 20 on the WB value of the previous frame. The image processing unit 20 performs pre-WB processing using the instructed WB value in step S100 described above. The WB value may be calculated within the image processing unit 20. In this case, the camera control unit 18 acquires the WB value to be used in the pre-WB processing from the camera control unit 18 in step S2.

[0066] In step S3, the camera control unit 18 generates metadata that sets the sensor array information and the WB value (pre-WB information) used in the pre-WB processing. In this case, setting them as a set means adding information indicating that the sensor array information and the pre-WB information are data to be used in the subsequent development processing of the YC-RAW data.

[0067] The sensor array information is information used to restore image data, which has been converted to the RGB plane state of FIG. 5 by demosaic processing, to the original sensor array state by re-mosaic processing, which will be described later. Therefore, the sensor array information may include information on the filter type that determines the pixel array and the first pixel, as exemplified in FIG. 4. For example, if the image sensor 12a uses the Bayer array primary color filter of FIG. 4, the camera control unit 18 configures the sensor array information to include information that the filter type is an "RGGB" type with three RGB primary colors and that the first pixel is "R." However, the sensor array information need only be information that enables identification of pixel values ​​output by the image sensor and interpolated pixel values ​​obtained by demosaic. The pre-WB information is used in the inverse pre-WB processing described below. In other words, in the inverse pre-WB processing, the WB value in the pre-WB processing is included in the metadata so that the inverse value of the WB value used in the pre-WB processing can be used to return to a state where no pre-WB processing has been performed.

[0068] 6, the camera control unit 18 associates the generated metadata, i.e., the set of sensor array information and WB value, with the YC-RAW data generated by the image processing unit 20. For example, when a still image is obtained as YC-RAW data by single-shot or continuous-shot capture as a still image, the WB value of the pre-WB processing and the sensor array information are associated with the YC-RAW data as the still image and recorded on a recording medium. Alternatively, the metadata is transmitted to an external device together with the YC-RAW data as the still image. When recording each frame of YC-RAW data as a video, the WB value of the pre-WB processing and the sensor array information are recorded in association with each frame. Note that since the sensor array information does not change for each frame, it may be associated with the entire video file of YC-RAW data. The sensor array information is used in the re-mosaic processing described later, but there are cases where it does not need to be associated with the YC-RAW data as metadata. For example, it is also possible that re-mosaic processing will be performed later by the image processing unit 20 in the imaging device 1. In this case, the sensor array information may be stored in advance in the memory unit 19 or the like in the imaging device 1. Even when re-mosaic processing is performed by the image processing unit 20 in the information processing device 70, if the information processing device 70 has correspondence data between information on the model of the imaging device 1 and the sensor array information, it is possible to obtain the sensor array information to be referenced during the re-mosaic processing.

[0069] FIG. 7 shows another example of the metadata generation process of the camera control unit 18. Steps S1 and S2 are the same as those in Fig. 6. In step S10, the camera control unit 18 acquires light source information. For example, the light source information may be estimated information on the type of light source in the shooting environment from information from the illuminance sensor in the sensor unit 23, or the color temperature itself.

[0070] In step S11, the camera control unit 18 generates metadata by combining light source information with the WB value and sensor array information of the pre-WB processing described in Figure 6, and adding information indicating that the data will be used in subsequent development processing of YC-RAW data. Then, in step S4, a set of sensor array information, WB value, and light source information is controlled to be recorded or transmitted in association with the YC-RAW data, as in the example of FIG.

[0071] In the imaging device 1, when YC-RAW data is generated by imaging, as in the processing of Figures 6 or 7 above, information to be used for subsequent inverse pre-WB processing, re-mosaic processing, or development processing of the YC-RAW data can be associated with the YC-RAW data as metadata.

[0072] <4. Comparative example of YC-RAW development> The development process for YC-RAW data will be described below, but first an example of development process will be described as a comparative example with reference to FIG. 8, which does not include the color shift correction in the embodiment described below.

[0073] It should be noted that the YC-RAW data development processing described below (comparison examples and embodiment examples) may be performed by the image processing unit 20 in the imaging device 1, or may be performed by the image processing unit 20 in the information processing device 70. Therefore, the term "control unit 100" will be used in the explanation of Fig. 8 and Figs. 10, 12, 14, etc., which will be described later. This "control unit 100" is a term that refers to both the camera control unit 18 in Fig. 1 and the CPU 71 in Fig. 2. In other words, the component that controls the image processing unit 20 in the imaging device 1 and the information processing device 70 is referred to as the "control unit 100."

[0074] The development process by the image processing unit 20 as a comparative example in FIG. 8 is carried out as follows. For example, YC-RAW data recorded on a recording medium is read out and input to the image processing unit 20 . The image processing unit 20 takes in the YC-RAW data that has been selected as the subject of development processing in the interface processing of step S200.

[0075] Because this YC-RAW data was generated by high-speed pre-WB processing during the YC-RAW imaging process described in FIG. 3, errors in the pre-WB must be eliminated. To achieve this, the image processing unit 20 performs RGB conversion processing on the YC-RAW data in step S201 to convert it back to the original image sensor output format, i.e., RGB format in this example, and then performs inverse pre-WB processing in step S202. This is followed by standard signal processing for development, namely, main WB processing in step S203, matrix conversion in step S204, gamma processing in step S205, and color reproduction processing in step S206. Finally, in interface processing in step S207, a developed image dPD, which is the image data obtained through the above development processing, is output. The developed image dPD may be, for example, JPEG image data. This developed image dPD may be recorded on a recording medium, displayed, or transmitted to an external device.

[0076] In the RGB conversion process in step S201, the YCbCr format is converted into the RGB format using a calculation formula that is the reverse of the format conversion used in the YC-RAW imaging process. Then, the inverse pre-WB processing unit in step S202 uses the inverse of the WB value used in the pre-WB processing during YC-RAW imaging processing to restore the RGB signal levels before the pre-WB processing, thereby canceling out the difference due to the pre-WB.

[0077] However, the signal from the image sensor 12a during YC-RAW imaging processing is a signal of intermittent data before demosaic processing, while the image data subjected to this inverse pre-WB processing is full-plane image data in RGB format after being interpolated by demosaic processing. Therefore, for the RGB plane image data converted from this YC-RAW data, even if the difference caused by the pre-WB processing can be offset by the inverse pre-WB processing, it is not possible to eliminate the effects of color fading caused by the demosaic processing.

[0078] In the main WB processing in step S203, the entire frame is detected again, and the WB for development is calculated, followed by highly accurate WB processing. The user may also specify a WB value based on their preference, development purpose, or application. In such cases, the WB value set by the user takes precedence. If the user does not specify a WB value, the WB value obtained by the high-precision detection for development described above is used. At this time, AE (Automatic Exposure) processing, that is, brightness adjustment, may be performed.

[0079] If a user-specified WB value is used here, the WB value may be intentionally set to a value that is significantly different from the ideal WB, which could potentially exacerbate the color shift caused by the demosaic processing described above.

[0080] For example, in the image shown in Figure 9A, the B and R value levels of each pixel in the area enclosed by the dashed line H are shown in Figures 9B and 9C, illustrating the state of color shift. Figure 9B shows the signal levels for the original image, while Figure 9C shows the signal levels for an image with color shift. In this example, the G value is omitted because it is low and the difference is small. Looking at the areas surrounded by dashed lines in Figures 9B and 9C, it can be seen that the R and B values ​​have decreased due to the differential effects of pre-WB processing and demosaic processing, resulting in a color shift from red to orange.

[0081] In the process shown in Figure 8, after this WB processing, matrix conversion, gamma processing, and color reproduction processing are performed in steps S204 to S206. Color reproduction processing includes color grading using 3DLUT or CDL, and also includes color space conversion. Furthermore, conversion to luminance and color difference signals (YCbCr signals) and resolution conversion may also be performed.

[0082] 8, the inverse pre-WB processing cannot completely eliminate the effects of color shifts caused by the pre-WB processing and demosaic processing when generating YC-RAW data. Therefore, the development processing of the embodiment described below is performed.

[0083] <5. First embodiment: YC-RAW development> The development processing of YC-RAW data in this embodiment is a development processing that includes a color shift correction processing that corrects color shifts caused by pre-WB processing and demosaic processing. The development process including the color shift correction process refers to a development process that includes at least the re-mosaic process described below. FIG. 10 shows an example of development processing by the image processing unit 20 in the same format as FIG.

[0084] The image processing unit 20 receives the YC-RAW data that has been subjected to development processing in the interface processing of step S200. For example, the YC-RAW data recorded on a recording medium is read out and input to the image processing unit 20. At this time, the image processing unit 20 also acquires metadata MT associated with the YC-RAW data. The metadata MT includes pre-WB information MD1, which is the WB value used in pre-WB processing, and sensor array information MD2 (see FIG. 6). It may also include light source information MD3 (see FIG. 7).

[0085] In step S201, the image processing unit 20 performs RGB conversion processing on the YC-RAW data to convert it into image data in the image sensor output format, that is, image data in the RGB format in this example. In step S220, the image processing unit 20 performs inverse pre-WB processing. At this time, the image processing unit 20 performs inverse pre-WB processing by referencing the pre-WB information MD1. That is, inverse pre-WB processing uses the inverse of the WB value of the pre-WB used during YC-RAW imaging processing to restore the RGB signal levels before pre-WB processing, thereby canceling out the difference due to pre-WB. However, as described above, this does not eliminate the effect of color fading due to demosaic processing during YC-RAW imaging processing.

[0086] Therefore, image processing unit 20 performs a re-mosaic process in step S221. This is a process of returning the image data after demosaic processing to the state of RAW data obtained from image sensor 12a by referring to sensor array information MD2. The image data before the re-mosaic process in step S221 is the RGB plane data shown in FIG. 5. By acquiring the first pixel and pixel array pattern as sensor array information MD2, it is possible to identify the pixel arrangement of each color as shown in FIGS. 11A, 11B, and 11C. Therefore, from the pixel data in the RGB plane, it is possible to restore RGB-RAW data with a pixel array in the original image sensor output format as shown in FIG. 11D. This is the re-mosaic process. In other words, it refers to the sensor array information MD2, discards interpolated pixels, and leaves only the pixel values ​​obtained as pixel signals from the image sensor 12a. By performing this re-mosaic processing, the information of the interpolated pixels is eliminated, and color shifts caused by the demosaic processing can be eliminated.

[0087] In step S222, the image processing unit 20 performs the main WB processing on the image data that has been subjected to the re-mosaic processing. To perform this main WB processing, the image processing unit 20 performs detection processing for the main WB in step S224. For example, full detection is performed using RGB format image data after re-mosaic processing. Based on this detection result, for example, the control unit 100 (or the image processing unit 20) sets a WB value for the main WB processing. Furthermore, if the user performs an operation to specify WB parameters as the WB value for the main WB, the WB value specified by the user is set preferentially.

[0088] If the metadata MT contains light source information MD3, the light source information MD3 can be referenced during the detection process in step S224, thereby enabling more accurate setting of the WB value.

[0089] The WB processing in step S222 is performed using a highly accurate WB value based on full detection or a WB value based on a user setting. Note that AE processing may also be performed at this time. In step S223, the image processing unit 20 performs demosaic processing again. In this demosaic processing, WB processing is performed appropriately, that is, with high precision or according to user settings, so that unintended WB deviations do not further increase.

[0090] In subsequent steps S204 to S207, matrix conversion processing, gamma processing, color reproduction processing, and interface processing are performed in the same manner as in the example of FIG. 8, and a developed image dPD is output.

[0091] As described above, the development process of the first embodiment refers to the pre-WB information MD1 and sensor array information MD2 in the metadata MT, and performs inverse pre-WB processing (S220) and re-mosaic processing (S221) on the demosaiced RGB image data converted from the YC-RAW data to restore it to the state of the RAW data immediately after it was input from the imaging unit 12. This eliminates the effects of color shift due to the demosaicing during YC-RAW image capture. Therefore, by subsequently performing the main WB processing (S222), followed by the demosaicing (S223), and then performing the matrix conversion processing (S204) and subsequent processing, a developed image dPD is obtained that does not exhibit the effects of color shift due to the demosaicing during YC-RAW image capture.

[0092] <6. Second embodiment: YC-RAW development> The development process of YC-RAW data according to the second embodiment will be described with reference to FIG. In recent years, post-processing has become popular, and there are an increasing number of cases where users specify WB. From a signal processing perspective, it is considered better for overall image quality if the subsequent matrix conversion and color reproduction are also adjusted in conjunction with WB. Furthermore, if the user's preferences are known in advance, it is desirable from a usability perspective to recommend development processes that match the user's preferences.

[0093] Therefore, it is conceivable to carry out development processing in multiple systems. In FIG. 12, the processes in steps S200, S201, S220, and S221, that is, the re-mosaic process, are the same as those in FIG. 10, so a duplicated description will be avoided. In the example of FIG. 12, the WB process and subsequent processes are performed as multiple systems.

[0094] For example, the development processing in steps S222, S223, S204, S205, S206, and S207 is the processing explained in Fig. 10. For example, in the main WB processing in step S222, WB processing is performed using a highly accurate WB value or a WB value based on a user setting. Then, after each processing, a developed image dPD (in this case, developed image dPD1) is finally output. This developed image dPD1 can be recorded on a recording medium, displayed, or transmitted to an external device.

[0095] On the other hand, steps S232 to S238 show the development process for the second system. In step S232, the image processing unit 20 performs second WB processing using a WB value different from that used in the main WB processing in step S222. For example, this may be a different WB value specified by the user or a WB value automatically set by the control unit 100. Then, in step S233, the image processing unit 20 performs demosaic processing, matrix conversion (S234), gamma processing (S235), and color reproduction processing (S236). The parameters for each of these processes are user-specified parameters or parameters set according to the WB value used in the second WB processing. As a result, a developed image dPD2 is generated as one of the developed images dPD, which has a different hue and image atmosphere from the developed image dPD1.

[0096] Note that the developed image dPD2 after step S236 may be output as is and recorded on a recording medium or transmitted to an external device, but in this example, difference processing is performed in step S237. As a difference process, the image processing unit 20 performs a difference calculation between the developed image dPD1 and the developed image dPD2 to generate difference information DD2. The difference information DD2 is output by the interface process of step S238. This difference information DD2 is associated with the above-mentioned developed image dPD1 and recorded on a recording medium or transmitted to an external device. When recording or transmitting multiple developed images dPD as a set, it is not necessary to record each developed image dPD as is; instead, it is sufficient to use difference information between one developed image and a reference developed image for the other developed images. For example, for developed image dPD2, the difference information DD2 is recorded or transmitted in association with developed image dPD1, rather than the developed image dPD2 itself. This allows the developed image dPD2 to be reproduced, and also reduces the data volume as a set of multiple developed images, which is advantageous for recording and communication.

[0097] Steps S242 to S248 show the development process for the third system. In step S242, the image processing unit 20 performs third WB processing using a WB value different from that used in the main WB processing and the second WB processing. For example, this may be a different WB value specified by the user or a WB value automatically set by the control unit 100. Then, in step S243, the image processing unit 20 performs demosaic processing, matrix conversion (S244), gamma processing (S245), and color reproduction processing (S246). The parameters for each of these processes are user-specified parameters or parameters set according to the WB value used in the third WB processing. As a result, a developed image dPD3 is generated as one of the developed images dPD, which has a different hue and image atmosphere from the developed images dPD1 and dPD2.

[0098] In this case, the developed image dPD3 may also be output as is and recorded on a recording medium or transmitted to an external device, but by performing differential processing in step S247, the size of the developed image set can be reduced in the same manner as above. As a difference process, the image processing unit 20 performs a difference calculation between the developed image dPD1 and the developed image dPD3 to generate difference information DD3. The difference information DD3 is output by the interface process in step S248. This difference information DD3 is recorded on a recording medium or transmitted to an external device.

[0099] The difference information DD3 may be the difference from the developed image dPD2 instead of the difference from the developed image dPD1. Furthermore, when multiple types of developed images dPD1, dPD2, and dPD3 are generated, it is not necessary to use the normally developed image dPD1 as the reference. There are various possibilities as to which developed image to use as the reference and which developed image to use as difference information.

[0100] 12 shows an example in which three systems of development processing are performed, but four or more systems of processing may be performed, or just two systems. By performing development processing in two or more systems, multiple types of developed images dPD can be obtained. For example, Figure 13 shows examples of developed images dPD1 obtained through normal development processing, as well as various developed images dPD2, dPD3, dPD4, and dPD5 obtained through other types of development processing. By preparing and selectively using parameter sets for WB value, matrix conversion, gamma processing, and color reproduction processing, developed images with a variety of image qualities can be obtained.

[0101] All of these multiple types of developed images may be generated and recorded or transmitted (hereinafter also referred to as "recording, etc."), or a developed image desired by the user may be generated and recorded, etc. When the user is allowed to select the developed image to be recorded, etc., it is also conceivable to recommend developed images to the user to record, etc. For example, it is conceivable to set parameters according to the conditions and intentions at the time of development (e.g., hard / soft contrast), multiple types of WB (auto WB, cloudy, sunny), or user preferences (popular styles such as personal or regional styles: red emphasis, high saturation), and present these to the user.

[0102] In this case, in order to make it easier for the user to understand, it is preferable to give a title of the image type to each of the parameter sets for obtaining each developed image as shown in FIG. For example, parameter sets for obtaining each image such as developed images dPD1, dPD2, dPD3, dPD4, and dPD5 are presented to the user with titles such as "normal development," "cloudy," "sunny," "monochrome," "high contrast," and "soft contrast." A model of image quality corresponding to each title may also be displayed, allowing the user to select a parameter set for the image quality of their choice. For example, if the user selects "normal development" or "cloudy weather" during development processing, the image processing unit 20 performs two-way development processing to obtain developed images dPD1 and dPD2.

[0103] Alternatively, the control unit 100 may automatically select a parameter set of "normal development," "sunny weather," or "monochrome" according to the user's preferences, history of past development processes, etc., recommend it at the time of development processing, and cause the image processing unit 20 to execute the type of development processing selected according to a user operation in response to the recommendation. Alternatively, the control unit 100 may control the image processing unit 20 to automatically execute multiple types of development processing selected according to the user's preferences, etc.

[0104] Multiple sets of image processing parameters may be stored in the metadata MT in association with the YC-RAW data when, for example, YC-RAW imaging processing is performed by the imaging device 1. During development processing, one or more parameter sets may be acquired by referring to the metadata MT and reflected in the development processing.

[0105] 7. Third Embodiment: Overwriting of YC-RAW As the third embodiment, a processing example will be described in which the image processing unit 20 overwrites and records the YC-RAW data to which color shift correction has been applied, over the original YC-RAW data.

[0106] Fig. 14 shows the processing of the image processing unit 20 in the same format as Fig. 10. The development processing for YC-RAW data up to steps S200, S201, S220, S221, S222, S223, S224, S204, S205, S206, and S207 is the same as in Fig. 10. Here, the image data that has undergone the re-mosaic processing in step S221 has been corrected for color shifts caused by the pre-WB processing and demosaic processing in the YC-RAW imaging processing, and has become data in RAW format in the original image sensor output format.

[0107] In step S208, the image processing unit 20 performs YC conversion processing on the image sensor output format data that has undergone this color shift correction, converting it into YCbCr format data, i.e., YC-RAW data. This will be referred to as "YC-RAW' data" to distinguish it from the YC-RAW data that is the target of development processing. The generated YC-RAW' data is output in the interface process of step S209. The control unit 100 causes this YC-RAW' data to be recorded on, for example, a recording medium. In the case of the imaging device 1, the YC-RAW' data is recorded on a recording medium by the recording control unit 14. In the case of the information processing device 70, the YC-RAW' data is recorded on a recording medium by the recording control unit 79.

[0108] At this time, the recording control unit 14 (or 74) overwrites and records the YC-RAW' data onto the recording medium from which the YC-RAW data to be developed has been read.

[0109] That is, the original YC-RAW data to be developed, which still contained the effects of color shift, is overwritten with YC-RAW' data in which the effects of color shift have been eliminated. As a result, the YC-RAW data (= YC-RAW' data) saved on the recording medium does not need to have color shift corrected in subsequent development processes. Therefore, this is a useful process when considering the reuse of YC-RAW data later. It should be noted that the YC-RAW' data may not be overwritten onto the original YC-RAW data, but may be recorded on a recording medium so as to coexist as color shift corrected data separate from the original YC-RAW data.

[0110] 14, YC-RAW' data is recorded, but the image data that has been subjected to the re-mosaic processing in step S221, i.e., the RAW format data in the original image sensor output format, may be recorded on the recording medium so as to overwrite the original YC-RAW data. Alternatively, the RAW format data in the original image sensor output format may be recorded on the recording medium so as to coexist with the original YC-RAW data.

[0111] Also, in FIG. 14, an example is shown in which the image processing unit 20 performs development processing in steps S222, S223, and S204 to S207 and also outputs the developed image dPD, but the image processing unit 20 may not generate and output the developed image dPD, but may only generate and output for recording YC-RAW' data or RAW format data in the image sensor output format.

[0112] 8. Fourth Embodiment: Processing with Determination of Color Misalignment Correction Execution So far, we have described how inverse pre-WB processing and re-mosaic processing are performed as color shift correction processes when developing YC-RAW data, and how main WB processing and demosaic processing are then performed. However, adding color shift correction processing every time YC-RAW data is developed may increase the amount of calculations on the image processing unit 20, increasing the processing load. Also, color shift correction processing may not always be necessary. Therefore, an example in which development processing including color shift correction processing is performed depending on the situation will be described as the fourth embodiment. In this case, it will also be described how the control unit 100 automatically determines whether color shift correction processing is necessary, and how the user can specify whether or not to perform color shift correction processing via a UI (user interface).

[0113] The user may specify whether or not to perform development processing including color shift correction processing in two ways: via a UI during image capture, or via a UI during development. When capturing an image, the user can specify the image by operating a menu, for example, and the specified information can be included in the metadata MT associated with the YC-RAW data, so that the control unit 100 can confirm the specified content when developing the image. Furthermore, during development, the control unit 100 can allow the user to select specified information through menu operations, or a UI can be considered in which the control unit 100 determines the need for color shift correction processing for individual images, and if it determines that it is necessary, allows the user to choose whether or not to perform the processing.

[0114] An example of processing performed by the control unit 100 during development processing is shown in Fig. 15. Fig. 15 shows an example of processing performed on one frame of YC-RAW data.

[0115] When one frame of YC-RAW data is to be subjected to development processing in the image processing unit 20, the control unit 100 determines in step S300 whether the YC-RAW data is set to include color shift correction. For example, if it is set at the time of image capture, the control unit 100 checks the metadata MT. Alternatively, the control unit 100 may check the setting set by the user during development.

[0116] The user can set whether or not to perform color misregistration correction processing using a UI such as a menu screen 40 as shown in Fig. 16. Fig. 16 shows an example of a menu screen displayed on the display unit 15 of the imaging device 1. Of course, a menu screen that allows similar settings to be made may also be displayed on the display unit 77 of the information processing device 70 in a format suitable for the screen.

[0117] For example, the menu screen 40 has a setting area 41 for "advanced color reproduction processing," allowing the user to specify various settings.

[0118] Advanced color reproduction processing is processing that includes the color shift correction processing described above and increases the accuracy of color reproduction during image capture. When advanced color reproduction processing is set to "ON," development processing that includes the color shift correction processing described in Figure 10 and elsewhere is performed. When advanced color reproduction processing is set to "OFF," development processing is set to not include the color shift correction processing described in Figure 8 (development processing that does not include re-mosaic processing).

[0119] Furthermore, the process for determining whether or not to perform advanced color reproduction processing can be selectively set to one of "high-speed," "high-precision," or "balanced." For example, Figure 16 shows a schematic diagram of the state in which "high-speed" is set. The high-speed type is a mode in which judgment processing is performed using an algorithm that can be executed relatively quickly, while the high-precision type is a mode in which judgment processing is performed using an algorithm with relatively high precision. Specific examples of each will be described later. The balanced type is a mode that automatically switches between high speed and high precision according to the image characteristics, etc.

[0120] In step S300 of FIG. 15, the control unit 100 checks the contents set by the user through operations on the menu screen 40 during image capture or development. If the control unit 100 detects from the metadata MT that the advanced color reproduction processing is "ON" for the current frame, or if it detects that the user has set the advanced color reproduction processing "ON" for the current development processing, the process proceeds from step S300 to step S310, and instructs the image processing unit 20 to perform development processing including color shift correction processing. As a result, a developed image dPD that has been subjected to color shift correction is obtained through the processing described above with reference to Figures 10, 12, 14, etc.

[0121] For example, by leaving the advanced color reproduction processing "ON," the user can avoid the effects of color shifts caused by pre-WB processing and demosaic processing, and can expect high image quality in the developed image dPD. However, the additional processing steps can result in longer processing times. If you want to shorten the processing time of the development process, you can set the advanced color reproduction process to "OFF." In this case, the judgment process will determine whether color shift correction processing is necessary, and the development process including color shift correction processing will be performed as necessary.

[0122] If the advanced color reproduction processing is set to "OFF" based on the user's operation during image capture or development, the control unit 100 proceeds from step S300 to step S301, and starts a determination process to determine whether color shift correction processing is necessary.

[0123] In step S301, the control unit 100 selects a determination algorithm. For example, a high-speed type or a high-precision type is selected. In this case, the selection may be made according to user settings on the menu screen 40 of FIG. If the high-speed type is set, a high-speed algorithm is selected. If the high-precision type is set, a high-precision algorithm is selected. If the balanced type is set, either the high-speed or high-precision algorithm is selected depending on the situation. For example, if one frame of YC-RAW data as a still image is to be developed, the high-precision type may be selected, and if multiple frames of YC-RAW data as a video are to be developed, the high-speed type may be selected.

[0124] In step S302, the control unit 100 branches the process depending on whether the selected algorithm is a high-speed type or a high-precision type. If the high-speed type is selected, the control unit 100 proceeds to step S303.

[0125] An example of a high-speed algorithm will be described. First, the WB value for this WB processing is calculated based on the RAW data input during development processing. At this time, if light source information MD3 is included in the metadata MT, this can also be used to calculate the WB value for this WB processing. Furthermore, if the processing target is YC-RAW data of multiple frames from continuous shooting or video, applying the WB average value of the past few frames to the current frame can reduce the time required to calculate the WB detection value and also improve WB stability.

[0126] After the WB value of the main WB processing is calculated, the difference ΔWB between the pre-WB and the main WB is calculated, and it is determined whether this difference is greater than a threshold value TH1. in particular, ΔWB = (WB value of pre-WB processing) - (WB value of main WB processing) If ΔWB>TH1, it is determined that there is a high possibility of color misregistration. On the other hand, if ΔWB≦TH1, it is determined that there is a low possibility of color misregistration. The threshold value TH1 can basically be set to a few percent or less of the pre-WB R gain and B gain levels, but it is desirable to adjust it in accordance with the actual image characteristics.

[0127] The above processing is the determination processing using the high-speed algorithm in steps S303 and S304. If it is determined in step S304 that ΔWB>TH1 is not true, the control unit 100 determines that color shift correction is not necessary, proceeds to step S309, and instructs the image processing unit 20 to perform development processing that does not include color shift correction processing as shown in Figure 8.

[0128] If it is determined in step S304 that ΔWB>TH1, the control unit 100 proceeds to step S307 and performs warning display control. For example, in the case of the imaging device 1, a warning display 45 such as that shown in Fig. 17 is displayed on the display unit 15. For example, the UI notifies the user that there is a high possibility that the image will have color shifts if development processing is performed as is, and allows the user to select whether or not to apply correction processing. In the case of the information processing device 70, the display unit 77 is caused to display a warning with the same content.

[0129] In response, the control unit 100 waits for a user operation, and if the user instructs application of color shift correction processing, the process proceeds from step S308 to step S310, and instructs the image processing unit 20 to perform development processing that includes color shift correction processing.If the user instructs not to apply color shift correction processing, the control unit 100 proceeds from step S308 to step S309, and instructs the image processing unit 20 to perform development processing that does not include color shift correction processing.

[0130] When a high-precision algorithm is selected by the selection of the determination algorithm in step S301, the control unit 100 proceeds from step S302 to step S305, and performs determination processing using the high-precision algorithm.

[0131] The high-precision algorithm, for example, observes the signal level after inverse pre-WB, and if the signal level is significantly changed by the difference ΔWB described above, determines that a color shift has occurred at the pixel in question. Specifically, the following color shift parameter CSL (CSL: Color Shift Level) is defined, and the determination is made based on the value of the color shift parameter CSL.

[0132] ΔCSL=ΔWB×MAX(RGB_i) "RGB_i" is the RGB signal value after inverse pre-WB in a certain block, and MAX(RGB_i) is the maximum value of the RGB signal value.

[0133] Then, using a threshold value TH2, it is determined that the possibility of color misregistration is high if ΔCSL>TH2, and low if ΔCSL≦TH2. The threshold value TH2 is basically set to a few percent of the signal level in the current block, but it is desirable to adjust it in accordance with the actual image characteristics.

[0134] The above processing is the determination processing using the high-precision algorithm in steps S305 and S306. If it is determined in step S306 that ΔCSL>TH2 is not true, the control unit 100 determines that color shift correction is not necessary, proceeds to step S309, and instructs the image processing unit 20 to perform development processing that does not include color shift correction processing as shown in Figure 8.

[0135] If it is determined in step S306 that ΔCSL>TH2, the control unit 100 proceeds to step S307 and performs warning display control. For example, in the case of the imaging device 1, a warning display 45 such as that shown in Fig. 18 is displayed on the display unit 15. For example, the UI notifies the user that there is a high possibility that the image will have color shifts if development processing is performed as is, and allows the user to select whether or not to apply correction processing. Furthermore, when a high-precision algorithm is applied, color shift determination is performed for each block within a frame, so a color shift area display 46 can be displayed. For example, the portion corresponding to the block where color shifts have occurred is presented to the user as a zebra pattern or the like. Of course, the zebra pattern is just one example, and the area where color shifts have occurred can be clearly indicated by highlighting, discoloring, or displaying a frame, for example.

[0136] Furthermore, the control unit 100 may at this stage cause the image processing unit 20 to execute processing that adds color shift correction processing, and perform control so that, as shown in Figure 19, the color shift corrected developed image 47 and the normal developed image 48 are displayed side by side on the same screen to present to the user the images before and after the elimination of color shift. For example, the screens of FIG. 18 and FIG. 19 may be switchable by a user operation, or the control unit 100 may automatically switch the display. This makes it easier for the user to select the development processing method.

[0137] After displaying warning 45, control unit 100 waits for a user operation, and if the user instructs application of color shift correction processing, the process proceeds from step S308 to step S310, instructing image processing unit 20 to perform development processing including color shift correction processing. If the user instructs not to apply color shift correction processing, control unit 100 proceeds from step S308 to step S309, instructing image processing unit 20 to perform development processing without color shift correction processing.

[0138] 15, when the determination process for determining the necessity of color shift correction processing is performed, an algorithm for the determination process is selected, and, as necessary, a warning display 45 or the like is executed. In other words, even if the advanced color reproduction processing is not set to ON on the menu screen 40 so that color shift correction processing is always applied, the user can optionally execute development processing with color shift correction processing applied if it is determined that color shift correction processing is necessary.

[0139] Although an example in which a warning display 45 is displayed has been given, other examples are also possible. For example, when it is determined in the determination process that color shift correction is necessary, detailed information about the color shift, such as coordinate information for the area where unnatural color shift has occurred and the ΔCSL value, may be recorded as metadata associated with the developed image dPD, so that it may be possible to select later whether or not to perform color shift correction processing.

[0140] Here, the selection of the determination algorithm in step S301 will be described. As described above, the specification of a high-speed or high-precision algorithm can be made in ways other than the user's specification through menu operations (prioritizing on high speed, high precision, or balanced). For example, the control unit 100 may be configured to automatically select a determination algorithm based on the camera settings. Note that the following automatic selection method may also be applied to selection when the balanced type is specified on the menu screen 40.

[0141] As an example of automatic selection of the judgment algorithm, -Selection by high-speed continuous shooting -Select with or without face detection Selection by power saving mode of the imaging device 1 Possible reasons include:

[0142] For example, the faster the continuous high-speed shooting speed, the lower the accuracy of the pre-WB processing. This is because there is less time to calculate the WB value, making it difficult to maintain accuracy. Therefore, the faster the continuous high-speed shooting speed, the greater the color shift tends to be. Therefore, one possible example is to separate the high-speed continuous shooting speed at a certain threshold speed, and apply a high-speed algorithm when the speed is faster than the threshold speed, and apply a high-precision algorithm when the speed is not higher.This is because, in images captured at a high-speed continuous shooting speed where it is difficult to maintain accuracy, a clear difference is likely to be detected as the difference ΔWB, and therefore accurate judgment can be expected even with a high-speed algorithm. For the same reason, it is also conceivable to apply a high-speed algorithm when developing YC-RAW data obtained by continuous shooting or video shooting, and a high-precision algorithm when developing YC-RAW data obtained by single-shot shooting.

[0143] The reason for the selection based on whether or not face detection is performed is as follows. When comparing an image in which a face is detected with an image in which no face is included, the effect of color shift is more noticeable when a face is included. Therefore, it is possible to apply a high-precision algorithm to frames in which a face is detected, and a high-speed algorithm to frames in which a face is not detected.

[0144] The power saving mode of the imaging device 1 is selected for the following reasons. A smaller amount of calculation is preferable when the remaining battery charge is low or when the power saving mode is ON in the imaging device 1. Therefore, it is conceivable to apply a high-speed algorithm with a smaller amount of calculation when the remaining battery charge is low or when the power saving mode is ON, and to apply a high-precision algorithm when such a situation is not the case.

[0145] Incidentally, when performing development processing including the color shift correction processing described in the first embodiment, high image quality can be expected, but the development processing time may increase. In particular, when the process in Figure 15 always proceeds from step S300 to step S310, the processing time tends to become longer. Therefore, it is possible to devise ways to shorten the processing time. For example, as shown in Figure 20, it is possible to shorten the processing time by dividing the image into blocks and performing pipeline processing. Also, during development, developed images dPD can be generated and displayed using a process that does not include the color shift correction process shown in Figure 8, while developed images dPD can be generated as background processing using a process that includes the color shift correction process shown in Figure 10, and the developed images dPD can be replaced sequentially. In this case, it is possible to prevent processing delays from being noticeable to the user.

[0146] As a processing example of the control unit 100 when performing the determination processing, a processing example as shown in FIG. 21 is also conceivable. Steps S300, S309, and S310 in FIG. 21 are the same as those in FIG. For example, if the advanced color reproduction processing is set to "ON" in the menu settings, development processing including the color shift correction processing described with reference to FIG. 10 and the like is performed in step S310. If the advanced color reproduction processing is set to "OFF," color shift determination processing is performed in step S320. In this case, determination processing is performed using a fixed determination algorithm. For example, determination processing is always performed using a high-speed algorithm in step S320. It is also possible to always perform determination processing using a high-precision algorithm.

[0147] If it is determined that there is a high possibility of color shift, the control unit 100 proceeds from step S321 to step S310 and instructs the image processing unit 20 to perform development processing that includes color shift correction processing.If it is determined that there is a low possibility of color shift, the control unit 100 proceeds from step S321 to step S309 and instructs the image processing unit 20 to perform development processing that does not include color shift correction processing.

[0148] As described above, the example in Fig. 21 is an example in which a fixed determination algorithm is used, and in which a development processing method is selected in accordance with the determination process without waiting for a user selection.

[0149] Furthermore, as for the determination process, it is possible to perform a determination using not only the above-mentioned determination algorithm based on signals obtained inside the image processing unit 20 and the control unit 100 but also the metadata MT. For example, the light source estimated in the pre-WB processing is determined from pre-WB information MD1 related to the pre-WB processing when performing YC-RAW imaging processing in the imaging device 1. Alternatively, the light source determination information is included in the pre-WB information MD1. On the other hand, there is also light source information MD3 estimated from the detection value of the illuminance sensor in the sensor unit 23. In this case, the possibility of color shift can be determined based on whether the light source determination information based on the pre-WB information MD1 matches the light source information MD3.

[0150] The processing example of FIG. 22 is an example in which such a determination process is used. In step S330, the control unit 100 determines whether the light source determination information based on the pre-WB information MD1 matches the light source information MD3.

[0151] If they do not match, the control unit 100 determines that there is a high possibility of color shift and that color shift correction is necessary, and proceeds from step S331 to step S310, instructing the image processing unit 20 to perform development processing that includes color shift correction processing.If the results of step S330 match, there is a low possibility of color shift and color shift correction processing is unnecessary, and the control unit 100 proceeds from step S321 to step S309, instructing the image processing unit 20 to perform development processing that does not include color shift correction processing.

[0152] This makes it possible to easily switch the development processing method, that is, whether or not to add color shift correction processing.

[0153] <9. Summary and Variations> According to the above embodiment, the following effects can be obtained. The image processing device of the embodiment, i.e., the image processing unit 20 (or the imaging device 1 or information processing device 70 including the image processing unit 20), performs development processing on YC-RAW data, which is image data obtained by performing pre-WB processing, demosaic processing, and YC conversion processing on image sensor output, for example, RGB format image data. In this case, conversion processing is performed to convert the YC-RAW data into the image sensor output format (for example, RGB format), and further performs inverse pre-WB processing of the pre-WB processing and re-mosaic processing to return the data to the state before demosaic processing. By performing color shift correction processing on YC-RAW data, including inverse pre-WB processing and re-mosaic processing, it is possible to eliminate color shifts caused by the pre-WB processing and demosaic processing performed when generating the YC-RAW data. Then, by developing that YC-RAW data, it is possible to prevent color shifts from occurring during development. This improves the image quality of the developed image dPD developed from the YC-RAW data. This also promotes the effective use of YC-RAW data by taking advantage of its advantages, such as its smaller data size than RGB-RAW data, less image degradation when resized, and its suitability for high-speed recording. Note that, although RGB format image data has been used as an example of image data in the image sensor output format, image data in the image sensor output format may also be image data in the Cy, Ye, G, Mg format based on the complementary color filter described in Figure 4, or image data in the WRGB format based on the W-RGB filter.

[0154] In the first, second and third embodiments, examples have been described in which the image processing unit 20 performs the main WB processing and demosaic processing on image data after re-mosaic processing (see FIGS. 10, 12 and 14). By performing reverse pre-WB processing and re-mosaic processing to eliminate the color shift contained in the original YC-RAW data, and then performing this WB processing and demosaic processing, RGB plane image data with no color shift from the state at the time of capture can be obtained. From this state, matrix conversion processing, gamma processing, color reproduction processing, etc. can be performed and development can be performed to obtain a high-quality developed image dPD with no color shift.

[0155] In the second embodiment, an example has been described in which the image processing unit 20 performs a plurality of development processes with mutually different parameter settings on the image data after the re-mosaic process to generate a plurality of developed images dPD. For example, multiple developed images dPD1, dPD2, dPD3, etc. are generated using different parameters for this WB processing, matrix conversion processing, gamma processing, and color reproduction processing. This makes it possible to provide users with developed images that have been created in a variety of ways. Users can select images according to their preferences, rather than simply images that are faithful to the color conditions at the time of capture.

[0156] In the second embodiment, an example has been described in which the image processing unit 20 generates difference information DD between a reference developed image and some of the developed images dPD. For example, when multiple developed images dPD1, dPD2, and dPD3 are generated, difference information DD2 and DD3 is generated between the developed images dPD2 and dPD3 and the reference developed image dPD1. By recording this difference information DD2 and DD3 on a recording medium, the required recording capacity can be significantly reduced compared to recording the developed images dPD2 and dPD3 as is.

[0157] In the fourth embodiment, an example has been described in which the control unit 100 performs a determination process for determining the need for color shift correction on YC-RAW data, and causes the image processing unit 20 to perform re-mosaic processing depending on the determination result. For example, YC-RAW data to be developed does not necessarily have a large color shift. When the control unit 100 performs a judgment process and determines that the color shift is somewhat large, the image processing unit 20 performs development processing with color shift correction, thereby obtaining a high-quality developed image dPD. Conversely, if there is not much color shift, processing such as re-mosaicing can be unnecessary, which is suitable for reducing the processing load and speeding up the development process.

[0158] In the fourth embodiment, the control unit 100 determines the necessity of color shift correction by using the difference (ΔWB) between the WB value calculated for the YC-RAW data and the WB value used in the pre-WB processing of the YC-RAW data. That is, this is determination processing using a high-speed algorithm. This makes it possible to easily detect color misalignment in image data. In particular, because this is internal signal processing, it is possible to detect color misalignment in real time without delay by comparing the signal difference immediately before and after processing.

[0159] In the fourth embodiment, an example was given in which the control unit 100 determines the necessity of color shift correction for YC-RAW data based on the amount of change ΔCSL in the signal level after inverse pre-WB processing. That is, this is determination processing using a high-precision algorithm. This is also internal signal processing, so the judgment can be made relatively quickly. In addition, by making the judgment for each pixel / block, the area where the color shift is occurring can be accurately identified.

[0160] In the processing example of Figure 22 in the fourth embodiment, an example is given in which the control unit 100 performs a judgment process to determine the need for color shift correction using information based on the WB value used in the pre-WB processing of the YC-RAW data and light source information detected at the time of image capture. The light source information MD3 can be used in the case of an imaging device 1 having a sensor that can obtain the light source information MD3, or in the case of an information processing device 70 that has acquired YC-RAW data with the light source information MD3 added to the metadata MT. In this case, the presence or absence of color shift can be estimated by comparing the light source information MD3 with the light source estimated by pre-WB processing. In this case, the determination process can be performed extremely simply and quickly.

[0161] In the processing example of FIG. 15 in the fourth embodiment, the control unit 100 is allowed to select a plurality of processing methods as the determination processing, and performs the determination processing using the selected processing method. For example, the processing method for the judgment process can be selectively executed by using a high-speed algorithm that can be executed relatively quickly or a high-precision algorithm that has relatively high accuracy, etc. This makes it possible to perform judgment processing that is appropriate for the situation.

[0162] In the processing example of FIG. 15 in the fourth embodiment, an example has been described in which the control unit 100 selects the processing method of the determination processing based on user settings. For example, the control unit 100 selects a determination method (algorithm) depending on whether the user has set a high-speed type, a high-precision type, or the like in a menu setting, etc. This enables determination processing according to the user's purpose, use case, etc.

[0163] In the processing example of FIG. 15 in the fourth embodiment, an example has been described in which the control unit 100 automatically selects the processing method for the determination processing. For example, the control unit 100 automatically selects a determination method (algorithm) such as a high-speed type or a high-precision type depending on some trigger, setting, imaging situation, etc. For example, the selection is made depending on the high-speed continuous shooting speed, the presence or absence of a face image, the battery situation, etc. This enables determination processing that is suitable for the imaging operation, image content, device situation, etc.

[0164] In the processing example of FIG. 15 in the fourth embodiment, an example has been described in which control is performed to issue a warning when the control unit 100 detects color misregistration through the determination process. For example, if the control unit 100 detects that color shift has occurred as a result of the determination process, it executes a warning display 45 and a color shift area display 46 (see FIGS. 17, 18, and 19). This allows the user to recognize that color shift has occurred.

[0165] In the processing example of FIG. 15 in the fourth embodiment, an example has been given in which the control unit 100 controls the image processing unit 20 to perform re-mosaic processing in response to a user operation in response to a warning (see steps S308 and S310 in FIG. 15). The user, who is made aware by the warning that color shift will occur, can choose whether or not to correct color shift in the development process, depending on his or her purpose, preference, convenience, etc.

[0166] In the first embodiment, an example has been given in which the image processing unit 20 performs inverse pre-WB processing using the pre-WB information MD1 in the metadata MT associated with the YC-RAW data. By adding the pre-WB information MD1 as the metadata MT, it is possible to accurately perform inverse pre-WB processing, which returns the pre-WB processing to the state before the processing.

[0167] In the first embodiment, an example has been given in which the image processing unit 20 performs re-mosaic processing using the sensor array information MD2 in the metadata MT associated with the YC-RAW data. By adding the sensor array information MD2 as metadata MT, the re-mosaic processing can be performed accurately.

[0168] In the first embodiment, an example was given in which the image processing unit 20 calculates the WB value of the actual WB using light source information MD3 in the metadata MT associated with the YC-RAW data to be developed, and then performs the actual WB processing. By adding light source information as metadata MT, this WB processing can be performed with values ​​that correspond to the light source at the time of image capture.

[0169] The image processing device described in the third embodiment is equipped with a recording control unit (14, 79) that performs conversion processing on YC-RAW data into an image sensor output format, and further performs inverse pre-WB processing on the pre-WB processing and re-mosaic processing to return the data to the state before demosaic processing, and records the image data in the image sensor output format obtained after that, or YC-RAW' data obtained based on the image data in the image sensor output format obtained after re-mosaic processing, on a recording medium. After applying color shift correction processing to YC-RAW data using inverse pre-WB processing and re-mosaic processing, the resulting YC-RAW' data is converted to YC and is free of the color shift caused by the pre-WB processing and demosaic processing that is present in the original YC-RAW data. By overwriting the original YC-RAW data with the resulting YC-RAW' data, the YC-RAW' data can then be used as RAW image data that can be used to develop high-quality images. The YC-RAW' data may also be recorded so that it coexists with the original YC-RAW data. Furthermore, data in a state where color shift correction processing has been performed on the YC-RAW data using inverse pre-WB processing and re-mosaic processing, i.e., data in the RAW format in the original image sensor output format, can be recorded on the recording medium together with the original YC-RAW data or overwritten. This data in the original image sensor output format can also be used as RAW image data for developing high-quality images. Furthermore, both the YC-RAW' data and the data in the image sensor output format may be recorded on the recording medium.

[0170] 1 includes an imaging unit 12 that obtains captured image data using an image sensor 12a, and an image processing unit 20 that generates YC-RAW data by performing pre-WB processing, demosaic processing, and YC conversion processing on the RGB format image data obtained by the imaging unit 12. The imaging device 1 also includes a camera control unit 18 that performs processing to associate sensor array information MD2 that represents the pixel array of the image sensor 12a as metadata MT used in re-mosaic processing that returns the image to the state before the demosaic processing, and pre-WB information MD1 that includes parameters for the pre-WB processing as metadata MT used in inverse pre-WB processing of the pre-WB processing, with the YC-RAW data (see FIGS. 6 and 7). By associating pre-WB information MD1 and sensor array information MD2 as metadata MT with YC-RAW data, it becomes possible to accurately perform reverse pre-WB processing and re-mosaic processing by referring to these, for example, during development processing.

[0171] Furthermore, the imaging device 1 may include an illuminance sensor as a sensor capable of detecting light source information. In this example, the camera control unit 18 processes the sensor array information MD2, the pre-WB information MD1, and the light source information MD3 generated based on the detection value of the sensor as metadata MT to be used in the development process of the YC-RAW data (see FIG. 7). By including light source information MD3 as metadata MT, more accurate main WB processing can be performed by referring to this information during main white balance processing during development processing.

[0172] The program relating to the development processing of the embodiment is a program that causes the development processing shown in FIGS. 10 and 12 to be executed by, for example, a CPU, DSP, GPU, GPGPU, AI processor, or a device including any of these. In other words, as a program for an embodiment, a program can be envisioned that performs conversion processing to RGB format on YC-RAW data, which is image data that has undergone pre-WB processing, demosaic processing, and YC conversion processing on RGB format image data, and further causes an arithmetic processing device that functions as an image processing device to perform inverse pre-WB processing on the pre-WB processing, and re-mosaic processing to return the image to the state before demosaic processing.

[0173] The program according to the embodiment is a program that causes a processing device to execute a process for generating YC-RAW' data as shown in FIG. In other words, as a program for an embodiment, a program can be envisioned that causes a processing device to execute a process of converting YC-RAW data, which is image data that has been subjected to pre-WB processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, into the original image sensor output format, and then recording the image data in the image sensor output format obtained after further performing inverse pre-WB processing on the pre-WB processing and re-mosaic processing to return it to the state before demosaic processing, on a recording medium. Alternatively, a program may be envisioned that causes a processing device to execute a process of recording onto a recording medium YC-RAW data obtained based on image data in the image sensor output format obtained after the above-mentioned re-mosaic processing.

[0174] By using such programs, the image processing device referred to in the present disclosure can be realized by various computer devices.

[0175] These programs can be recorded in advance on a HDD as a recording medium built into a device such as a computer, or on a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0176] Furthermore, such a program is suitable for providing the image processing device of the present disclosure to a wide range of devices. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the image processing device of the present disclosure. A computer device that functions as the image processing device of the present disclosure using such a program can also be realized as a cloud server. For example, the processes described in the embodiments can be executed by cloud computing in cooperation with the imaging device 1.

[0177] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0178] The present technology can also be configured as follows. (1) The image processing unit performs a conversion process to the image sensor output format for YC-RAW data, which is image data that has been subjected to pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, and further performs an inverse pre-white balance process for the pre-white balance process and a re-mosaic process that returns the image to the state before the demosaic process. Image processing device. (2) The image processing unit performs white balance processing and demosaic processing on the image data after the re-mosaic processing. The image processing device according to (1) above. (3) The image processing unit performs a plurality of development processes with mutually different parameter settings on the image data after the re-mosaic process to generate a plurality of developed images. The image processing device according to (1) or (2) above. (4) The image processing unit generates difference information between a part of the plurality of developed images and a reference developed image. The image processing device according to (3) above. (5) A control unit performs a determination process to determine the need for color shift correction for the YC-RAW data, and causes the image processing unit to perform the re-mosaic process according to the determination result. The image processing device according to any one of (1) to (4) above. (6) The control unit performs the determination process by: The necessity of correcting color shift is determined using the difference between the white balance value calculated for the YC-RAW data and the white balance value used in the pre-white balance processing for the YC-RAW data. The image processing device according to (5) above. (7) The control unit performs the determination process by: The necessity of correcting color shift is determined for the YC-RAW data based on the amount of change in the signal level after the inverse pre-white balance processing. The image processing device according to (5) or (6) above. (8) The control unit performs the determination process by: The necessity of color shift correction is determined using information based on the white balance value used in the pre-white balance processing of the YC-RAW data and light source information detected during image capture. The image processing device according to any one of (5) to (7) above. (9) The control unit is capable of selecting a plurality of processing methods as the determination processing, and performs the determination processing using the selected processing method. The image processing device according to any one of (5) to (8) above. (10) The control unit selects a processing method for the determination process based on a user setting. The image processing device according to (9) above. (11) The control unit automatically selects a processing method for the determination processing. The image processing device according to (9) above. (12) The control unit performs control to issue a warning when it is determined that color misregistration correction is necessary by the determination process. The image processing device according to any one of (5) to (11) above. (13) The control unit controls the image processing unit to perform the re-mosaic processing in response to a user operation in response to the warning. The image processing device according to (12) above. (14) The image processing unit performs the inverse pre-white balance processing using pre-white balance information in metadata associated with the YC-RAW data. An image processing device according to any one of (1) to (13) above. (15) The image processing unit performs the re-mosaic processing using sensor array information in metadata associated with the YC-RAW data. An image processing device according to any one of (1) to (14) above. (16) In the development process, the image processing unit performs the white balance process after the re-mosaic process using light source information in metadata associated with the YC-RAW data to be developed. The image processing device according to (2) above. (17) a recording control unit that performs a conversion process to the image sensor output format for YC-RAW data, which is image data that has been subjected to pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, and further performs an inverse pre-white balance process for the pre-white balance processing and a re-mosaic process for returning the image to the state before the demosaic processing, and records on a recording medium the image data in the image sensor output format obtained after the re-mosaic processing, or the YC-RAW data obtained based on the image data in the image sensor output format obtained after the re-mosaic processing Image processing device. (18) an imaging unit that obtains captured image data using an image sensor; an image processing unit that performs pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format obtained by the imaging unit to generate YC-RAW data; a control unit that performs processing to associate sensor array information indicating a pixel array of the image sensor with YC-RAW data as metadata used in a re-mosaic process that returns the image to the state before the demosaic process, and pre-white balance information including parameters for the pre-white balance process as metadata used in an inverse pre-white balance process for the pre-white balance process. Imaging device. (19) a sensor capable of detecting light source information; The control unit performs processing to generate light source information based on the detection value of the sensor as metadata to be used in development processing of YC-RAW data. The image processing device according to (18) above. (20) YC-RAW data is image data that has undergone pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, and is then converted into the image sensor output format. Furthermore, reverse pre-white balance processing is performed on the pre-white balance processing, and re-mosaic processing is performed to return the image to the state before the demosaic processing. Image processing methods. [Explanation of symbols]

[0179] 1. Imaging device 12 Imaging unit 12a image sensor 14 Recording control section 15 Display 18 Camera control unit 20 Image processing section 23 Sensor section 40 Menu screen 41 Setting area 45 Warning display 46 Color misalignment area display 70 Information processing device, 71 CPU 77 Display section 78 Audio output section 79 Recording control section 100 control section

Claims

1. The image processing unit performs a conversion process to the image sensor output format for YC-RAW data, which is image data that has been subjected to pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, and further performs an inverse pre-white balance process for the pre-white balance process and a re-mosaic process that returns the image to the state before the demosaic process. Image processing device.

2. The image processing unit performs white balance processing and demosaic processing on the image data after the re-mosaic processing. The image processing device according to claim 1 .

3. The image processing unit performs a plurality of development processes with mutually different parameter settings on the image data after the re-mosaic process to generate a plurality of developed images. The image processing device according to claim 1 .

4. The image processing unit generates difference information between a part of the plurality of developed images and a reference developed image. The image processing device according to claim 3 .

5. A control unit performs a determination process for determining whether or not color shift correction is necessary for the YC-RAW data, and causes the image processing unit to perform the re-mosaic process according to the determination result. The image processing device according to claim 1 .

6. The control unit performs the determination process by: The necessity of correcting color shift is determined using the difference between the white balance value calculated for the YC-RAW data and the white balance value used in the pre-white balance processing for the YC-RAW data. The image processing device according to claim 5 .

7. The control unit performs the determination process by: Regarding the YC-RAW data, the necessity of color shift correction is determined based on the amount of change in the signal level after the inverse pre-white balance processing. The image processing device according to claim 5 .

8. The control unit performs the determination process by: The necessity of color shift correction is determined using information based on the white balance value used in the pre-white balance processing for the YC-RAW data and light source information detected during image capture. The image processing device according to claim 5 .

9. The control unit is capable of selecting a plurality of processing methods as the determination processing, and performs the determination processing using the selected processing method. The image processing device according to claim 5 .

10. The control unit selects a processing method for the determination process based on a user setting. The image processing device according to claim 9 .

11. The control unit automatically selects a processing method for the determination processing. The image processing device according to claim 9 .

12. The control unit performs control to issue a warning when it is determined that color misregistration correction is necessary by the determination process. The image processing device according to claim 5 .

13. The control unit controls the image processing unit to perform the re-mosaic processing in response to a user operation in response to the warning. The image processing device according to claim 12.

14. The image processing unit performs the inverse pre-white balance processing using pre-white balance information in metadata associated with the YC-RAW data. The image processing device according to claim 1 .

15. The image processing unit performs the re-mosaic processing using sensor array information in metadata associated with the YC-RAW data. The image processing device according to claim 1 .

16. The image processing unit performs the white balance processing after the re-mosaic processing using light source information in metadata associated with the YC-RAW data. The image processing device according to claim 2 .

17. a recording control unit that performs a conversion process into the image sensor output format for YC-RAW data, which is image data that has been subjected to pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, and further performs an inverse pre-white balance process for the pre-white balance processing and a re-mosaic process for returning the image to the state before the demosaic processing, and records on a recording medium the image data in the image sensor output format obtained after the re-mosaic processing, or the YC-RAW data obtained based on the image data in the image sensor output format obtained after the re-mosaic processing Image processing device.

18. an imaging unit that obtains captured image data using an image sensor; an image processing unit that performs pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format obtained by the imaging unit to generate YC-RAW data; a control unit that performs processing to associate sensor array information indicating a pixel array of the image sensor with YC-RAW data as metadata used in a re-mosaic process that returns the image to the state before the demosaic process, and pre-white balance information including parameters for the pre-white balance process as metadata used in an inverse pre-white balance process for the pre-white balance process. Imaging device.

19. a sensor capable of detecting light source information; The control unit performs processing to generate light source information based on the detection value of the sensor as metadata to be used in development processing of YC-RAW data. The imaging device according to claim 18.

20. YC-RAW data, which is image data that has undergone pre-white balance processing, demosaic processing, and YC conversion processing on image data in an image sensor output format, is subjected to conversion processing into the image sensor output format, and further subjected to inverse pre-white balance processing for the pre-white balance processing and re-mosaic processing to return to the state before the demosaic processing. Image processing methods.

Images