Image processing device, image processing method, and program

Abstract

To provide an image processing device and an image processing method capable of generating RAW image files taking into consideration the performance differences of applications used in development processing. An image processing device performs image processing on a first RAW image to generate a second RAW image, and generates a RAW image file containing the second RAW image in one of a plurality of formats including a first format and a second format. The image processing device applies different image processing settings to the first RAW image when generating a RAW image file in the first format and when generating a RAW image file in the second format.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Image processing for the purpose of improving image quality may be applied to RAW format image data (hereinafter referred to as RAW image) obtained by an image sensor before it is stored in a RAW image file.

[0003] However, some information is lost when image processing is applied. Patent Document 1 discloses a technique for improving the image quality of a RAW image that has undergone image processing during development processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-114265 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the image quality improvement capability during development processing differs depending on the application used for the development processing.

[0006] The present invention has been made in view of the above-mentioned problems, and in one aspect thereof, provides an image processing device and an image processing method that are capable of generating a RAW image file that takes into account the differences in the capabilities of applications used in development processing. [Means for solving the problem]

[0007] In one aspect, the present invention provides an image processing device comprising: an image processing means for performing image processing on a first RAW image to generate a second RAW image; a generation means for generating a RAW image file including the second RAW image in one of a plurality of formats including a first format and a second format; and a control means for controlling the operation of the image processing means, wherein the control means changes the image processing settings when the generation means generates a RAW image file in the first format and when the generation means generates a RAW image file in the second format. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that can generate a RAW image file that takes into account the difference in capabilities of applications used in development processing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device as an example of an image processing device according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of a configuration in which a first RAW image is processed by an external device. [Figure 3] FIG. 3 is a block diagram showing an example of the functional configuration of a general-purpose computer capable of implementing the image processing system shown in FIG. 2. [Figure 4] 1 is a flowchart showing an example of a process for generating a RAW image file and a process for developing the generated RAW image file. [Figure 5] A flowchart showing a series of processes related to RAW image file generation and development processing when a first RAW image in a manufacturer-specific format RAW image file is included. [Figure 6] Flowchart regarding the operation of a dedicated application that executes development processing DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, the present invention will be described in terms of an embodiment using an imaging device as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smartwatches, game consoles, robots, drones, drive recorders, etc. Note that these are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] <Configuration and Functions of the Imaging Device> The configuration and functions of an image capture device 100 as an example of an image processing device according to the present invention will be described with reference to FIG.

[0013] The imaging device 100 can communicate with an external device (not shown) via the communication unit 13. The imaging device 100 can transmit, for example, an image file recorded on the recording medium 10 or an image file generated by the compression / decompression circuit 9 to the external device.

[0014] The imaging optical system 1 has a plurality of lens groups and an aperture mechanism 18. The plurality of lens groups include, for example, a zoom lens 16 and a focus lens 17. The imaging optical system 1 forms an optical image of a subject on the imaging surface of an imaging element 2.

[0015] The image sensor 2 converts the subject image that has passed through the imaging optical system 1 into an electrical signal. The image sensor 2 may be, for example, a CCD or CMOS image sensor. In this embodiment, the image sensor 2 includes an A / D converter that converts analog signals into digital signals, and is capable of outputting a digital RAW image. Note that the image sensor 2 does not include an A / D converter, and A / D conversion may be performed externally (e.g., by the image processing circuit 4). The image sensor 2 has a pixel array in which multiple pixels are arranged two-dimensionally. The pixel array is provided with, for example, a primary color Bayer array color filter. The image sensor 2 photoelectrically converts the subject optical image formed on the imaging surface by the imaging optical system 1 into a pixel signal group (hereinafter, referred to as an analog image signal) using multiple pixels. The image sensor 2 generates a RAW image by A / D converting the analog image signal and stores the RAW image in the memory 3.

[0016] The memory 3 stores images (including audio in the case of moving images) generated by the imaging element 2 and images to be displayed on the display unit 11. The memory 3 also stores images being processed by the compression / expansion circuit 9 and the image processing circuit 4, as well as images after processing. The memory 3 has a storage capacity sufficient to store a predetermined number of still images and moving images (including audio) for a predetermined period of time.

[0017] The image processing circuit 4 (image processing means) applies various image processing according to the purpose to the image stored in the memory 3. Image processing that the image processing circuit 4 can apply includes, for example, pre-processing, color interpolation processing, correction processing, detection processing, and data processing. Note that these are examples of image processing that the image processing circuit 4 can perform and do not limit the image processing that the image processing circuit 4 performs. The RAW image to which the image processing circuit 4 has applied image processing is saved in the memory 3.

[0018] The image processing applied to the image by the image processing circuit 4 also includes image processing for generating evaluation values ​​and signals used by the system control unit 5 for automatic exposure control (AE) and automatic focus detection (AF). The image processing circuit 4 can, for example, calculate luminance information of a specific area in the image as an evaluation value for AE. Furthermore, for example, if the image sensor 2 is capable of outputting an image for focus detection, the image processing circuit 4 can generate a signal used for phase difference detection AF from the image for focus detection and calculate the defocus amount using a known method. The image processing circuit 4 supplies the evaluation values ​​and signals for AE and AF to the system control unit 5.

[0019] The system control unit 5 (control means) is a control unit made up of at least one processor and / or at least one circuit, and controls the entire imaging device 100. The system control unit 5 loads a program recorded in the nonvolatile memory 15 into the system memory 14 and executes it, thereby controlling the operation of each unit of the imaging device 100 and realizing the functions of the imaging device 100. The system control unit 5 also mainly executes the processing described below using flowcharts.

[0020] The system control unit 5 performs AE and AF based on the evaluation value generated by the image processing circuit 4. Specifically, the system control unit 5 determines the exposure conditions (aperture value, shutter speed, sensitivity) based on the evaluation value for AE so that the focus detection area is properly exposed. Based on the exposure conditions, the system control unit 5 drives the aperture mechanism 18 via the exposure amount control unit 6 and also controls the operation of the image sensor 2. Based on the evaluation value for AF or the defocus amount calculated by the image processing circuit 4, the system control unit 5 drives the focus lens 17 via the focus lens control unit 7, thereby performing AF to focus the imaging optical system 1 on the subject.

[0021] As one of the operations in response to the operation of the operation member 12, the system control unit 5 drives the zoom lens 16 via the focal length control unit 8 to change the angle of view of the imaging optical system 1.

[0022] The system control unit 5 also controls the storage, reading and deletion of data on the recording medium 10 .

[0023] The system control unit 5 also functions as a generating unit that generates a RAW image file containing a RAW image. The system control unit 5 temporarily stores the generated RAW image file in the memory 3, and then saves it on the recording medium 10.

[0024] The system control unit 5 can generate RAW image files in multiple file formats. The file format is specified by the file name extension. In this specification, a RAW image refers to image data that has not undergone one or more of the image processing processes that depend on the device that captured the RAW image (here, the imaging device 100).

[0025] Image processing that depends on the imaging device includes, for example: - Processing to correct the characteristics of the image sensor and imaging optical system Demosaic (DeBayer) processing Pixel interpolation (defective pixel correction) processing White balance adjustment Edge enhancement processing Gamma processing These are examples, and other processes may be included.

[0026] Furthermore, the RAW image stored in the RAW image file may or may not have undergone image processing to improve image quality in the image processing circuit 4. Hereinafter, a RAW image to which image processing to improve image quality has not been applied will be referred to as a first RAW image, and a RAW image to which image processing to improve image quality has been applied will be referred to as a second RAW image. The RAW image file generated by the system control unit 5 may store either or both of the first RAW image and the second RAW image.

[0027] The system control unit 5 generates a RAW image file in one or more file formats according to, for example, user settings from among a plurality of file formats that can be generated. The type of RAW image (one or more of the first RAW image and the second RAW image) stored in the generated RAW image file may or may not depend on the file format. For example, even if a file format can store both the first RAW image and the second RAW image, only one of them may be included.

[0028] The exposure amount control unit 6 drives the aperture mechanism 18 of the imaging optical system 1 in accordance with instructions from the system control unit 5. The exposure amount control unit 6 also adjusts the drive timing and gain of the imaging element 2 in accordance with instructions from the system control unit 5.

[0029] The focus lens control unit 7 drives the focus lens 17 in the optical axis direction of the imaging optical system 1 and controls its position in accordance with instructions from the system control unit 5. This allows the imaging optical system 1 to be focused on the subject.

[0030] The focal length control unit 8 drives the zoom lens 16 in accordance with instructions from the system control unit 5. This changes the focal length (angle of view) of the imaging optical system 1.

[0031] The compression / decompression circuit 9 reads the image stored in the memory 3 and performs image compression and decompression processing by applying encoding and decoding in accordance with a predetermined encoding method. The compression / decompression circuit 9 also has a function of generating an image file that stores encoded data. The compression / decompression circuit 9 may support multiple encoding methods for both still images and moving images. The encoded data, image file, and decoded data generated by the compression / decompression circuit 9 are stored in the memory 3.

[0032] The compression / decompression circuit 9 saves the image file stored in the memory 3 on the recording medium 10 in accordance with instructions from the system control unit 5. The decoded data stored in the memory 3 can be converted by the image processing circuit 4 into image data for display and displayed on the display unit 11.

[0033] The display unit 11 displays the display image data stored in the memory 3 on a display device included in the imaging device 100 or an external display device.

[0034] The operation member 12 is one or more input devices that can be operated by a user, such as a switch, a button, or a touch panel. When the system control unit 5 detects an operation of the operation member 12, it executes an operation corresponding to the detected operation. The touch panel may be formed integrally with a display device included in the imaging device 100 or an external display device.

[0035] The communication unit 13 is a communication interface with an external device and supports one or more wired and wireless communication standards. The system control unit 5 can transmit, for example, image files recorded on the recording medium 10, encoded data generated by the compression / decompression circuit 9, and the like to the external device via the communication unit 13.

[0036] The system memory 14 is, for example, a RAM, and temporarily stores programs executed by the system control unit 5, constants and variables required for executing the programs, settings of the imaging device 100, and the like.

[0037] The nonvolatile memory 15 may be electrically erasable and recordable, such as an EEPROM, and stores information specific to the image capture device 100, user settings, GUI data, programs executed by the system control unit 5, and the like.

[0038] The second RAW image described above may be generated by the imaging device 100, or may be generated by a device separate from the imaging device 100. In addition, the development process of the RAW image may also be performed by the imaging device 100, or may be generated by a device separate from the imaging device 100. 2 is a diagram schematically illustrating an example of a series of image processing steps in which the imaging device 100 generates a RAW image file that includes a first RAW image but does not include a second RAW image, and the generation and development processing of the second RAW image are performed by devices separate from the imaging device 100. In FIG. 2, the process of generating the second RAW image from the first RAW image and the process of applying development processing to the second RAW image are performed by different devices (image processing systems), but they may also be performed by the same device.

[0039] In FIG. 2, the first image processing system 20 and the second image processing system 22 can be realized by executing an image processing application on an information processing device such as a personal computer.

[0040] FIG. 3 is a block diagram showing an example of the functional configuration of an information processing apparatus that can be used as the first and second image processing systems 20 and 22 in FIG.

[0041] The display 401 is, for example, a liquid crystal display (LCD), and displays a screen provided by the programs (OS and application programs) being executed by the CPU 405, etc.

[0042] The display controller 402 has a video RAM (VRAM) and controls the display on the display 401 .

[0043] The keyboard 403 and the pointing device 404 are examples of input devices that can be operated by a user. The user can operate the keyboard 403 and the pointing device 404 to input characters and instructions to the information processing device, and to operate GUI parts such as icons and buttons included in the screen displayed on the display 401.

[0044] The CPU 405 loads a program stored in the ROM 406 into the RAM 407 and executes it to realize the operation of the image processing system described below.

[0045] The ROM 406 stores programs executed by the CPU 405, various parameters, setting values, etc. The ROM 406 may be electrically rewritable. The RAM 407 is used as a work area when the CPU 405 executes various programs, a buffer for various data, etc.

[0046] The mass storage device 408 and the removable media drive (RMD) 409 are external storage devices. The mass storage device 408 may be a hard disk drive (HDD) or a solid state drive (SSD). The RMD 409 is a device that reads and writes data from or to a removable recording medium such as a memory card.

[0047] In addition, programs that realize various functions provided by the computer device, as well as application programs such as the OS and a web browser, data, libraries, etc., are stored in one or more of the ROM 406 and the mass storage device 408 depending on the application.

[0048] The expansion slot 410 can accommodate various expansion boards such as a video capture board and a sound board.

[0049] The network interface 411 is an interface for connecting the information processing device to a local network or an external network, and complies with one or more wired and wireless communication standards.

[0050] In addition to the network interface 411, the information processing device may also have a communication interface for communicating with external devices that conforms to various standards such as USB, HDMI (registered trademark), and Bluetooth (registered trademark).

[0051] The bus 412 comprises an address bus, a data bus, and a control bus, and connects the above-mentioned functional blocks.

[0052] The first and second image processing systems 20 and 22 may be any electronic device capable of executing an application program, such as a smartphone or a tablet.

[0053] The operations of the first and second image processing systems 20 and 22 described below are realized by the CPU 405 of each system executing an application program stored in the ROM 406 or the mass storage device 408.

[0054] First, we will explain the operation of the first image processing system 20. The first image processing system 20 acquires a RAW image file containing a first RAW image 19 generated by the imaging device 100. The first image processing system 20 can acquire the RAW image file from the imaging device 100 through communication between the network I / F 411 and the communication unit 13, for example, or from the recording medium 10 attached to the RMD 409.

[0055] The first image processing system 20 performs image processing on the first RAW image 19 to improve the image quality, and generates a second RAW image 21. The first image processing system 20 then generates a RAW image file including the second RAW image 21 and stores it in, for example, the mass storage device 408. The image processing performed on the first RAW image by the first image processing system 20 may or may not be the same as the image processing performed on the first RAW image by the image processing circuit 4 of the imaging device 100.

[0056] The second image processing system 22 acquires a RAW image file including a second RAW image 21 from the first image processing system 20, for example, through communication with the first image processing system 20 via the network I / F 411. The second image processing system 22 performs a development process on the second RAW image 21 to generate a developed image 23. The second image processing system 22 stores the generated developed image 23 in, for example, the mass storage device 408, or transmits it to an external device via the network I / F 411. The development process includes at least image processing dependent on the imaging device, and may also include image processing independent of the imaging device.

[0057] Image processing that is independent of the imaging device includes, for example, Exposure compensation processing Color correction processing Sharpness correction processing Retouching (image editing) processing Color space conversion processing Encoding process Scaling or cropping These are examples, and other processes may be included.

[0058] The developed image 23 can be used with various devices because the processing dependent on the imaging device has been completed. The developed image 23 has a general-purpose image file format such as JPEG format, HEIF format, or TIFF format.

[0059] In this embodiment, in the device (imaging device 100 or first image processing system 20) that generates the second RAW image, the image processing that is performed on the first RAW image to improve the image quality varies depending on the format of the RAW image file to be generated. Here, noise reduction processing is described as a typical example of image processing that improves image quality. However, the image processing that improves the image quality of the first RAW image may include one or more other image processing such as processing that corrects the effects of aberrations in the imaging optical system and demosaic (deBayer) processing.

[0060] RAW image file formats can be broadly divided into general-purpose formats, such as the DNG format, and formats specific to each imaging device manufacturer. Details of the data structure and stored data of manufacturer-specific RAW image files are not made public. Therefore, RAW image files in manufacturer-specific formats are generally intended to be processed using a development processing application (dedicated application) provided by that manufacturer. Although there are general-purpose development processing applications that are compatible with manufacturer-specific RAW image files, they may not be able to use all of the information contained in manufacturer-specific RAW image files. Furthermore, the extent to which the information contained in manufacturer-specific RAW image files is used may vary depending on the general-purpose development processing application.

[0061] On the other hand, generic RAW image files are intended to be processed by generic development applications (general-purpose applications). Furthermore, when a RAW image file is created, it is not possible to determine the capabilities of the generic application that will develop the RAW image file. Therefore, generic RAW image files tend to contain less information or less detailed information than manufacturer-specific RAW image files.

[0062] Therefore, the quality of the developed image 23 generated from a RAW image file using a general-purpose application in particular depends heavily on the capabilities of the general-purpose application. Therefore, in this embodiment, the image processing applied to the first RAW image is controlled according to the type or file format of the RAW image file to be generated, in order to reduce the impact that the capabilities of the general-purpose application that applies the development processing have on the image quality of the developed image 23.

[0063] 4 is a flowchart showing several examples of steps for generating a developed image after applying image processing to a first RAW image to improve its image quality. The flowchart shows three cases in which the development processing is performed by the image capture device 100, by an external device using a dedicated application, and by an external device using a general-purpose application. While FIG. 4 shows demosaic processing, edge enhancement processing, and gamma processing as common development processing, various other processing steps such as defective pixel correction and white balance adjustment, as described above, may also be performed.

[0064] Steps S100 to S106 are processes executed by the imaging device 100. In S100, the system control unit 5 sets the image processing to be applied to the first RAW image in the image processing circuit 4, depending on the format of the RAW image file to be generated. The relationship between the RAW image file format and the image processing settings can be stored in advance in the non-volatile memory 15. Note that this relationship may be changeable by the user. The system control unit 5 sets different image processing settings for a file format intended for development processing by a dedicated application or the image processing circuit 4, and for a file format intended for development processing by a general-purpose application. Details of the settings in S100 will be described later.

[0065] In S101, the image processing circuit 4 applies image processing to the first RAW image to improve image quality according to the settings made in S100, thereby generating a second RAW image. As an example, the image processing circuit 4 applies noise reduction processing to the first RAW image. The noise reduction processing can be performed by any known method. For example, the noise reduction processing may be performed using a trained neural network or a spatial low-pass filter.

[0066] In S102, the system control unit 5 generates a RAW image file in a file format according to the settings and saves it on the recording medium 10. Here, the system control unit 5 generates a RAW image file that stores the second RAW image regardless of the file format, but it may also generate a RAW image file that does not store the second RAW image.

[0067] Here, whether or not to store the first RAW image in the generated RAW image file is determined depending on the format of the RAW image file. Specifically, the system control unit 5 stores the first RAW image in a RAW image file generated in a manufacturer-specific format (first format), and does not store the first RAW image in a RAW file generated in a general-purpose format (second format). Note that it is not essential to store the first RAW image in a RAW file generated in the first format.

[0068] The execution of S102 and the execution of the next steps (S103, S107, S112) may be discontinuous in time. In S103 to S105, the image processing circuit 4 executes development processing on the second RAW image stored in the RAW image file generated in S102. First, in S103, the image processing circuit 4 executes demosaic processing on the second RAW image. Demosaic processing is a process in which missing color component values ​​for each pixel are interpolated based on the values ​​of surrounding pixels so that each pixel constituting the RAW image has each RGB color component. In this embodiment, the image sensor 2 has color filters in a primary color Bayer array, so demosaic processing is also called de-Bayer processing. After de-Bayer processing, the image processing circuit 4 converts the signal format of each pixel from RGB format to YCrCb format.

[0069] In S104, the image processing circuit 4 performs edge enhancement processing on the luminance component Y of the image after the DeBayer processing, thereby improving the perceived resolution of the image.

[0070] In S105, the image processing circuit 4 performs gamma processing on the edge-enhanced image. Gamma processing is also called gradation correction processing, and is processing that adjusts the gradation changes (light and dark and contrast) of the image. The image processing circuit 4 stores the gamma-processed image in the memory 3.

[0071] In S106, the compression / decompression circuit 9 applies encoding processing according to a preset developed image format to the gamma-processed image stored in memory 3 as needed, to generate developed image A. Developed image A has a general-purpose image file format such as JPEG, HEIF, or TIFF. The compression / decompression circuit 9 stores developed image A on recording medium 10.

[0072] S107 to S111 are development processes applied to the second RAW image stored in the RAW image file by a dedicated application provided by the manufacturer of the imaging device 100, which is executed by, for example, the second image processing system 22 in FIG. Furthermore, steps S112 to S117 are development processes applied to the second RAW image stored in the RAW image file by a general-purpose application executed by, for example, the second image processing system 22 in Fig. 2. S114 indicates a case where the general-purpose application can execute image processing for improving image quality as part of the development process. Whether or not S114 is included depends on the general-purpose application. Note that, for convenience, S107 and S112 for extracting a second RAW image to which development processing is to be applied from the RAW image file are described as part of the development processing, but strictly speaking, they are not included in the development processing.

[0073] Development processing by a development application is generally performed in accordance with interactive operations via a GUI screen provided by the application. Therefore, the order and number of processes performed by the dedicated application and the general-purpose application in the development processing are arbitrary. In other words, the user can adjust the parameters of each process by repeatedly setting each process and checking the results via the GUI screen provided by the development processing application. When the user instructs to save the developed image, the CPU 405 generates a developed image based on the parameters at that time and saves it, for example, in the mass storage device 408 (S111, S117).

[0074] The de-Bayering process performed by the dedicated application in S108, the edge enhancement process performed in S109, and the gamma process performed in S110 are performed using the same algorithm as the image processing circuit 4 in order to suppress differences with the developed image obtained by the imaging device 100.

[0075] On the other hand, the specific methods of the de-Bayering process performed by the general-purpose application in S113, the edge enhancement process performed in S115, and the gamma process performed in S116 may differ depending on the application. Therefore, even if the second RAW image stored in the RAW image file is the same, developed image B generated by the dedicated application and developed image C generated by the general-purpose application may differ.

[0076] Here, a specific example of image processing settings in S100 according to the file format will be described. The image processing applied in S101 may result in the loss of some of the information contained in the first RAW image. For example, consider the case where noise reduction processing is applied in S101 to reduce noise contained in the first RAW image and improve the image quality of the first RAW image. The noise reduction processing is image smoothing or low-pass filtering. The high-frequency components removed from the first RAW image by the noise reduction processing include not only noise components but also high-frequency components inherent in the image, which may result in a reduction in perceived resolution.

[0077] The device that generates the second RAW image (the imaging device 100 or the first image processing system 20; hereinafter referred to as the imaging device 100) typically has knowledge of the details of the development process applied by the dedicated application. Therefore, the device that generates the second RAW image can estimate the content and level of the image quality improvement process that will be applied in the development process by the dedicated application when generating the second RAW image. On the other hand, the device that generates the second RAW image cannot estimate the strength of the image quality improvement process that will be applied in the development process by the general-purpose application.

[0078] Therefore, in S100, the imaging device 100 (system control unit 5) The first case generates a RAW image file in a file format (manufacturer-specific format) intended for development processing in a dedicated application, and the second case generates a RAW image file in a file format (manufacturer-specific format) intended for development processing in a dedicated application. The second case generates RAW image files in a file format intended for development processing in a general-purpose application. The image processing circuit 4 varies the settings of the image processing applied to the first RAW image. For example, if the image processing applied to the first RAW image is noise reduction processing, the imaging device 100 (system control unit 5) varies the setting of the intensity of the noise reduction processing, for example.

[0079] For example, in the first case, the system control unit 5 can expect an improvement in resolution during development processing, so it can strengthen the intensity of noise reduction processing compared to the second case, in which the content and level of development processing are unpredictable.

[0080] Alternatively, in the second case where the content and extent of the development process cannot be predicted, the system control unit 5 may increase the strength of the noise reduction process compared to the first case to ensure that the effects of the noise reduction process are obtained, since there is a possibility that noise reduction process will not be performed during the development process.

[0081] Which is preferable may vary dynamically depending on the user and the intended use of the image, and so the system control unit 5 can set image processing according to the file format in accordance with user settings.

[0082] The intensity of the noise reduction process can be adjusted, for example, by the amount of high-frequency components added by texture correction performed on the RAW image after the noise reduction process. The image processing circuit 4 can obtain the high-frequency components removed by the noise reduction process, for example, by calculating the difference between the first RAW image before the noise reduction process and the second RAW image after the noise reduction process. Texture correction is a process for adjusting the resolution or noise amount by adding (restoring) some of the high-frequency components removed by the noise reduction process to the second RAW image. Texture correction can essentially weaken the effect of the noise reduction process. The system control unit 5 can control the intensity of the noise reduction process, for example, by controlling the proportion of high-frequency components added to the second RAW image by the image processing circuit 4 through texture correction. Note that if an intensity parameter can be set for the noise reduction process applied by the image processing circuit 4, the system control unit 5 can control the intensity of the noise reduction process by controlling the intensity parameter.

[0083] Furthermore, the system control unit 5 may change whether or not to store the first RAW image in the RAW image file between the first case and the second case. Specifically, the system control unit 5 can store the first RAW image in the RAW image file in the first case, and not store the first RAW image in the RAW image file in the second case.

[0084] Alternatively, in the first case, when the dedicated application can use the first RAW image during development processing, the system control unit 5 may store the first RAW image in a RAW image file.

[0085] FIG. 5 is a flowchart showing a series of processes in which the system control unit 5 generates a RAW image file in a manufacturer-specific format that includes a first RAW image, and develops the generated RAW image file using a dedicated application.

[0086] S100 and S101 are the same as in Fig. 4. In S102', the system control unit 5 generates a RAW image file in a manufacturer-specific format that includes the RAW image before the image processing in S101 was applied (first RAW image) and the RAW image after the image processing in S101 was applied (second RAW image). The system control unit 5 stores the RAW image file on the recording medium 10.

[0087] S201 and S108 to S111 are processes that are implemented by, for example, the second image processing system 22 (CPU 405) executing a dedicated application. First, the CPU 405 reads the first RAW image and the second RAW image from the RAW image file stored on the recording medium 10 and loads them into the RAM 407.

[0088] Then, the CPU 405 applies texture correction to the second RAW image using the first RAW image and the second RAW image. The texture correction can be performed using the same algorithm as the image processing circuit 4 of the imaging device 100, thereby reducing differences from the developed image in the imaging device 100. The strength of the texture correction in S201 may be fixed or may be determined interactively with the user. That is, the user can repeatedly set the strength of the texture correction and check the results through a GUI screen provided by the dedicated application to determine the final strength of the texture correction. Note that the order of the processes in S201 to S110 in the dedicated application is arbitrary, and the same processes may be repeatedly performed in response to a user instruction. Therefore, the user may readjust the strength of the texture correction based on the results of the edge enhancement (sharpness) process in S109 and the gamma process in S110.

[0089] Note that even if the image processing applied to the first RAW is other than noise reduction processing, such as optical aberration correction or de-Bayering, the settings can be varied depending on the file format, just as in the case of noise reduction processing. Specifically, in the case of optical aberration correction, the correction strength can be varied, and in the case of de-Bayering, the interpolation method (for example, the number and type of surrounding pixels used for interpolation) can be varied.

[0090] As described above, according to this embodiment, the image processing settings applied to the RAW image stored in the RAW image file are varied depending on the file format of the generated RAW image file. This makes it possible to apply image processing that takes into account the differences in the capabilities of the applications used for development processing, which are predicted from the file format, and increases the likelihood of obtaining an intended developed image with various development processing applications.

[0091] (Variation) If the dedicated application also supports development processing of RAW image files in a general format, the content of the development processing can be changed depending on the format of the RAW image file to be developed.

[0092] For example, consider a case where a first RAW image is included in a RAW image file in a manufacturer-specific format, and the first RAW image is not included in a RAW image file in a generic format. In this case, the CPU 405 executing the dedicated application can execute the process shown in the flowchart of FIG.

[0093] In S601, the CPU 405 determines whether the format of the RAW image file to be developed is a manufacturer-specific format or a generic format. This determination can be made, for example, based on the file extension. If the CPU 405 determines that the RAW image file to be developed is in a manufacturer-specific format, it executes S602, and if it determines that the RAW image file is in a generic format, it executes S603.

[0094] In S602, the CPU 405 determines to perform texture correction processing using the first RAW image as resolution or noise amount adjustment processing to be performed in the development processing.

[0095] In S603, the CPU 405 determines to perform processing that does not use the first RAW image (for example, dither processing) as the processing for adjusting the resolution or noise amount to be performed in the development processing.

[0096] Even if the first RAW image is not included in the RAW image file in the manufacturer-specific format, the content of the development process can be changed depending on the format of the RAW image file to be developed.

[0097] Although the above description is of an example in which the content of the resolution or noise amount adjustment process performed in the development process is changed, the content of other processes can also be changed. For example, the algorithms for de-Bayering, edge enhancement, gamma processing, etc. may be changed. Specifically, if the format is manufacturer-specific, the same algorithm as the development process executed in the imaging device can be used, and if the format is general-purpose, a different algorithm can be used.

[0098] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0099] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) an image processing means for performing image processing on the first RAW image to generate a second RAW image; a generating means for generating a RAW image file including the second RAW image in one of a plurality of formats including a first format and a second format; a control means for controlling the operation of the image processing means, The image processing device is characterized in that the control means changes the image processing settings when the generation means generates a RAW image file in the first format and when the generation means generates a RAW image file in the second format. (Item 2) 2. The image processing device according to item 1, wherein the image processing is image processing for improving image quality. (Item 3) 3. The image processing device according to item 1 or 2, characterized in that the image processing includes at least one of noise reduction processing, processing to correct the influence of aberration of the lens used to capture the first RAW image, and de-Bayer processing. (Item 4) 4. The image processing device according to item 3, wherein the image processing includes the de-Bayer processing. (Item 5) 4. The image processing device according to item 3, wherein the image processing does not include the de-Bayer processing. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the control means sets different intensity of the image processing when the generation means generates a RAW image file in the first format and when the generation means generates a RAW image file in the second format. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the RAW image file in the first format includes the first RAW image, and the RAW image file in the second format does not include the first RAW image. (Item 8) 7. The image processing device according to any one of items 1 to 6, wherein the RAW image file in the first format and the RAW image file in the second format include the first RAW image. (Item 9) Item 9. The image processing device according to item 7 or 8, characterized in that the control means, when the generation means generates a RAW image file in the first format, strengthens the intensity of the image processing compared to when the generation means generates a RAW image file in the second format. (Item 10) Item 9. The image processing device according to item 7 or 8, characterized in that when the generating means generates a RAW image file in the first format, the control means increases or decreases the intensity of the image processing compared to when the generating means generates a RAW image file in the second format. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the format of the RAW image file generated by the generating means is determined according to a user setting. (Item 12) 12. The image processing device according to any one of items 1 to 11, wherein the image processing device is an imaging device that generates the first RAW image. (Item 13) A means for acquiring a RAW image file; a developing means for applying a development process to the RAW image included in the RAW image file, The image processing device, wherein the developing means applies different development processes to the RAW images depending on the format of the RAW image file. (Item 14) Item 14. The image processing device according to item 13, characterized in that, when the RAW image file includes both a first RAW image and a second RAW image to which image processing has been applied to the first RAW image, the developing means applies processing using the first RAW image in the development processing for the second RAW image. (Item 15) Item 15. The image processing device according to item 14, wherein the processing using the first RAW image is a processing for adjusting resolution or noise amount. (Item 16) An image processing method performed by an image processing device, an image processing step of performing image processing on the first RAW image to generate a second RAW image; a generating step of generating a RAW image file containing the second RAW image in one of a plurality of formats including a first format and a second format; a control step for controlling the operation of the image processing step, The image processing method is characterized in that the control step differs the image processing settings when the generation step generates a RAW image file in the first format and when the generation step generates a RAW image file in the second format. (Item 17) An image processing method executed by an image processing device, an acquisition step of acquiring a RAW image file; a development step of applying development processing to the RAW image included in the RAW image file, The image processing method, wherein the developing step varies the development processing applied to the RAW image depending on the format of the RAW image file. (Item 18) 13. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 12. (Item 19) 16. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 13 to 15.

[0100] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0101] 100: Imaging device, 4: Image processing circuit, 5: System control unit, 10: Recording medium, 13: Communication unit, 19: First RAW image, 21: Second RAW image

Claims

1. an image processing means for performing image processing on the first RAW image to generate a second RAW image; a generating means for generating a RAW image file containing the second RAW image in one of a plurality of formats including a first format and a second format; a control means for controlling the operation of the image processing means, The image processing device is characterized in that the control means changes the image processing settings when the generation means generates a RAW image file in the first format and when the generation means generates a RAW image file in the second format.

2. 2. The image processing device according to claim 1, wherein the image processing is image processing for improving image quality.

3. 2. The image processing device according to claim 1, wherein the image processing includes at least one of noise reduction processing, processing for correcting the influence of aberration of a lens used to capture the first RAW image, and de-Bayer processing.

4. 4. The image processing apparatus according to claim 3, wherein the image processing includes the de-Bayering process.

5. 4. The image processing apparatus according to claim 3, wherein the image processing does not include the de-Bayering process.

6. 2. The image processing device according to claim 1, wherein the control means sets different image processing intensity settings when the generation means generates a RAW image file in the first format and when the generation means generates a RAW image file in the second format.

7. 2. The image processing apparatus according to claim 1, wherein the RAW image file in the first format includes the first RAW image, and the RAW image file in the second format does not include the first RAW image.

8. The image processing device according to claim 1 , wherein the first format RAW image file and the second format RAW image file contain the first RAW image.

9. 8. The image processing device according to claim 7, wherein the control means increases the intensity of the image processing when the generation means generates a RAW image file in the first format compared to when the generation means generates a RAW image file in the second format.

10. 8. The image processing device according to claim 7, wherein the control means weakens the intensity of the image processing when the generation means generates a RAW image file in the first format compared to when the generation means generates a RAW image file in the second format.

11. 2. The image processing apparatus according to claim 1, wherein the format of the RAW image file generated by said generating means is determined according to a user setting.

12. The image processing device according to claim 1 , wherein the image processing device is an imaging device that generates the first RAW image.

13. an acquisition means for acquiring a RAW image file; a developing means for applying a development process to the RAW image included in the RAW image file, The image processing device, wherein the developing means varies the development processing to be applied to the RAW image depending on the format of the RAW image file.

14. 14. The image processing device according to claim 13, wherein when the RAW image file includes both a first RAW image and a second RAW image obtained by applying image processing to the first RAW image, the developing means applies processing using the first RAW image in the development processing for the second RAW image.

15. 15. The image processing device according to claim 14, wherein the processing using the first RAW image is a processing for adjusting a sense of resolution or an amount of noise.

16. An image processing method performed by an image processing device, an image processing step of performing image processing on the first RAW image to generate a second RAW image; a generating step of generating a RAW image file containing the second RAW image in one of a plurality of formats including a first format and a second format; a control step for controlling the operation of the image processing step, An image processing method characterized in that the control step differs the image processing settings when the generation step generates a RAW image file of the first format and when the generation step generates a RAW image file of the second format.

17. An image processing method executed by an image processing device, an acquisition step of acquiring a RAW image file; a development process of applying development processing to the RAW image included in the RAW image file, The image processing method, wherein the developing step varies the development processing applied to the RAW image depending on the format of the RAW image file.

18. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 12.

19. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 13 to 15.

Images