Image processing apparatus and image processing method, and imaging apparatus

Abstract

To provide an image processing device that can assist in appropriately setting the parameters of image processing to be applied to image data that has already undergone image processing by another device. [Solution] The image processing device acquires a RAW image and additional information. Based on the additional information, the image processing device determines whether or not the first image processing has been applied to the RAW image, and determines the settings for the second image processing to be applied to the RAW image according to the result of the determination. Based on the determined settings, the image processing device applies the second image processing to the RAW image. If the image processing device determines that the first image processing has been applied to the RAW image, it determines the settings for the second image processing so that the intensity of the second image processing is weaker than when it is not determined that the first image processing has been applied to the RAW image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus, and particularly to an image processing apparatus and an image processing method for handling RAW images.

Background Art

[0002] Image data obtained using an imaging element is converted through various image processes into image data representing an image recognizable by humans as a photograph of a shooting scene (Patent Document 1). And a plurality of image processes applied in this conversion process are not necessarily applied by a single apparatus. There may be a case where, after storing image data to which a part of a plurality of image processes has been applied by a first apparatus, the remaining image processes are applied by a second apparatus. Also, the order and apparatus for applying image processes are not constant.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, among a plurality of image processes, there are some that have a relationship in which the application order affects the quality of the image represented by the image data after conversion. However, when two image processes having such a relationship are applied by separate apparatuses, it has been difficult to appropriately set the parameters of the image process to be applied later.

[0005] In view of such problems of the prior art, in one aspect of the present invention, there are provided an image processing apparatus and an image processing method capable of assisting in appropriately setting parameters of an image process to be applied to image data to which an image process has been applied by another apparatus.

Means for Solving the Problems

[0006] In one aspect, the present invention provides an image processing apparatus comprising: acquisition means for acquiring a RAW image and additional information; determination means for determining whether or not a first image processing has been applied to the RAW image based on the additional information; determination means for determining the settings for a second image processing to be applied to the RAW image according to the result of the determination; and processing means for applying the second image processing to the RAW image based on the settings determined by the determination means, wherein the determination means determines the settings for the second image processing such that, if it is determined that the first image processing has been applied to the RAW image, the intensity of the second image processing is weaker than when it is not determined that the first image processing has been applied to the RAW image. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing apparatus and an image processing method that can assist in appropriately setting the parameters of image processing to be applied to image data to which image processing has been applied by another device. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an example of the functional configuration of an imaging device as an example of an image processing apparatus according to an embodiment of the present invention. [Figure 2] A flowchart illustrating examples of the processes for generating RAW image files and developing the generated RAW image files. [Figure 3] A flowchart illustrating alternative methods for generating RAW image files and developing the generated RAW image files. [Figure 4] Figure 2 shows a block diagram illustrating an example of the functional configuration of a general-purpose computer capable of implementing the image processing system. [Modes for carrying out the invention]

[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] In the following description, the present invention will be implemented using an imaging device as an example of an image processing apparatus. However, imaging functionality is not essential to the present invention, and it can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include computer equipment (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smartwatches, game consoles, robots, drones, and dashcams. These are merely examples, and the present invention can be implemented in other electronic devices as well.

[0011] <Configuration and Function of Imaging Device> Referring to Figure 1, the configuration and function of an imaging device 100, which is an example of an image processing apparatus according to the present invention, will be described.

[0012] The imaging device 100 can communicate with an external device (not shown) via the communication unit 13. The imaging device 100 can transmit, for example, image files recorded on the recording medium 10 or image files generated by the compression / decompression circuit 9 to the external device.

[0013] The imaging optical system 1 has multiple lens groups and an aperture mechanism 18. The multiple lens groups include, for example, a zoom lens 16 and a focus lens 17. The imaging optical system 1 forms an optical image of the subject on the imaging surface of the image sensor 2.

[0014] The image sensor 2 converts the subject image that has passed through the imaging optical system 1 into an electrical signal. The image sensor 2 may be, for example, a CCD or a CMOS image sensor. In this embodiment, the image sensor 2 includes an A / D converter that converts the analog signal into a digital signal, and is capable of outputting a RAW image in digital format. Alternatively, the image sensor 2 may not include an A / D converter, and A / D conversion may be performed externally (for example, by an image processing circuit 4). The image sensor 2 has a pixel array in which multiple pixels are arranged in a two-dimensional array. The pixel array is provided with, for example, a color filter with a primary color Bayer array. The image sensor 2 converts the optical image of the subject formed on the imaging surface by the imaging optical system 1 into a group of pixel signals (hereinafter referred to as analog image signals) by photoelectric conversion using multiple pixels. The image sensor 2 generates a RAW image by A / D conversion of the analog image signal and stores it in the memory 3.

[0015] Memory 3 stores images (including audio in the case of video) generated by the image sensor 2, and images to be displayed on the display unit 11. Memory 3 also stores images being processed by the compression / decompression circuit 9 and the image processing circuit 4, as well as images after processing. Memory 3 has sufficient storage capacity to store a predetermined number of still images and a predetermined amount of video (including audio).

[0016] The image processing circuit 4 (image processing means) applies various image processing operations to the image stored in memory 3, depending on the purpose. Examples of image processing operations that the image processing circuit 4 can apply include preprocessing, color interpolation, correction, detection, and data processing. These are merely examples of image processing operations that the image processing circuit 4 can perform, and do not limit the image processing operations it can perform. The RAW image processed by the image processing circuit 4 is stored in memory 3.

[0017] In addition, the image processing applied by the image processing circuit 4 to the image includes image processing for generating evaluation values and signals used by the system control unit 5 for automatic exposure control (AE) and automatic focus detection (AF). The image processing circuit 4 can calculate, for example, the luminance information of a specific region in the image as an evaluation value for AE. Further, for example, when the imaging device 2 can output an image for focus detection, the image processing circuit 4 can generate a signal used for the phase difference detection method AF from the image for focus detection and calculate the defocus amount by a known method. The image processing circuit 4 supplies evaluation values and signals for AE and AF to the system control unit 5.

[0018] The system control unit 5 (control means) is a control unit composed of at least one processor and / or at least one circuit, and comprehensively controls the entire imaging device 100. The system control unit 5 reads the program recorded in the non-volatile memory 15 into the system memory 14 and executes it to control the operations of each part of the imaging device 100 and realize the functions of the imaging device 100. The system control unit 5 mainly executes the processes described later using the flowchart.

[0019] The system control unit 5 executes AE and AF based on the evaluation values generated by the image processing circuit 4. Specifically, the system control unit 5 determines exposure conditions (aperture value, shutter speed, sensitivity) such that the focus detection region has proper exposure based on the evaluation value for AE. Then, the system control unit 5 drives the aperture mechanism 18 through the exposure amount control unit 6 and also controls the operation of the imaging device 2 based on the exposure conditions. The system control unit 5 performs AF to focus the imaging optical system 1 on the subject by driving the focus lens 17 through the focus lens control unit 7 based on the evaluation value for AF or the defocus amount calculated by the image processing circuit 4.

[0020] As one of the operations corresponding to the operation on the operation member 12, the system control unit 5 drives the zoom lens 16 through the focal length control unit 8 to change the angle of view of the imaging optical system 1.

[0021] The system control unit 5 also controls the storage, reading, and deletion of data with respect to the recording medium 10.

[0022] The system control unit 5 also functions as generation means for generating a RAW image file including a RAW image. The system control unit 5 temporarily stores the generated RAW image file in the memory 3 and then stores it in the recording medium 10.

[0023] The system control unit 5 can generate RAW image files in a plurality of file formats. The file format is specified by the extension of the file name. In this specification, a RAW image refers to image data in a state where one or more image processes dependent on the device that captured the RAW image (here, the imaging device 100) have not been performed.

[0024] Image processes dependent on the imaging device include, for example · Processes for correcting image quality degradation caused by the characteristics of the imaging element and the imaging optical system (noise reduction processing, processing for correcting the influence of optical aberration, processing for correcting peripheral vignetting, etc.) · Demosaic (debayer) processing · Pixel interpolation (defective pixel correction) processing · White balance adjustment · Edge enhancement processing · Gamma processing These are examples, and other processes may be included.

[0025] Also, the RAW image stored in the RAW image file may be one to which one or more of a plurality of image processes dependent on the imaging device 100 have been applied by the image processing circuit 4, or one to which no image process has been applied. Hereinafter, a RAW image to which no image process has been applied is called a first RAW image, and a RAW image to which a part of the image process dependent on the imaging device 100 has been applied is called a second RAW image. The RAW image file generated by the system control unit 5 may store one or both of the first RAW image and the second RAW image.

[0026] The system control unit 5 generates a RAW image file in one or more file formats from among several possible file formats, for example, according to user settings. The type of RAW image stored in the generated RAW image file (one or more of a first RAW image and a second RAW image) may or may not depend on the file format. For example, the file format may be capable of storing both a first RAW image and a second RAW image, or it may contain only one of them.

[0027] The exposure control unit 6 drives the aperture mechanism 18 of the imaging optical system 1 according to the instructions of the system control unit 5. The exposure control unit 6 also adjusts the drive timing and gain of the image sensor 2 according to the instructions of the system control unit 5.

[0028] The focus lens control unit 7 drives the focus lens 17 in the optical axis direction of the imaging optical system 1 and controls its position according to the instructions of the system control unit 5. This allows the imaging optical system 1 to focus on the subject.

[0029] The focal length control unit 8 drives the zoom lens 16 according to the instructions of the system control unit 5. This changes the focal length (angle of view) of the imaging optical system 1.

[0030] The compression / decompression circuit 9 reads the image stored in memory 3 and performs image compression and decompression processing by applying encoding and decoding compliant with a predetermined encoding scheme. The compression / decompression circuit 9 also has a function to generate an image file that stores the encoded data. The compression / decompression circuit 9 may support multiple encoding schemes for both still images and videos. The encoded data, image file, and decoded data generated by the compression / decompression circuit 9 are stored in memory 3.

[0031] The compression / decompression circuit 9 saves the image file stored in memory 3 to the recording medium 10 according to the instructions of the system control unit 5. The decoded data stored in memory 3 is converted into display image data by the image processing circuit 4 and can be displayed by the display unit 11.

[0032] The display unit 11 displays the display image data stored in the memory 3 on a display device of the imaging device 100 or an external display device.

[0033] The operating component 12 is one or more user-operable input devices, such as switches, buttons, or touch panels. When the system control unit 5 detects an operation of the operating component 12, it executes an action corresponding to the detected operation. The touch panel may be integrally formed with the display device of the imaging device 100 or an external display device.

[0034] The communication unit 13 is a communication interface with external devices and supports one or more wired and wireless communication standards. The system control unit 5 can transmit, for example, image files recorded on the recording medium 10, encoded data generated by the compression / decompression circuit 9, etc., to external devices through the communication unit 13.

[0035] The system memory 14 is, for example, RAM. The system memory 14 temporarily stores programs executed by the system control unit 5, constants and variables necessary for program execution, settings for the imaging device 100, and so on.

[0036] The non-volatile memory 15 may be electrically erasable and recordable, such as an EEPROM. The non-volatile memory 15 stores unique information of the imaging device 100, user settings, GUI data, programs executed by the system control unit 5, and the like.

[0037] The second RAW image described above may be generated by the imaging device 100, or it may be generated by a device separate from the imaging device 100. Similarly, the development process for the RAW image may be performed by the imaging device 100, or it may be performed by a device separate from the imaging device 100.

[0038] Image processing is a series of image processing steps to convert RAW image data into image data compliant with common image formats such as JPEG, HEIF, and TIFF. In addition to the image processing dependent on the imaging device mentioned above, image processing also includes image processing independent of the imaging device, such as encoding according to the converted image format. Image data that has undergone image processing dependent on the imaging device can be used on various devices because the processing dependent on the imaging device has been completed.

[0039] Image processing that is independent of imaging equipment includes, for example, • Exposure compensation processing • Color correction processing • Sharpness correction processing • Retouching (image editing) • Color space conversion processing • Encoding process • Scaling or trimming These may include the following. These are examples, and other processes may be included.

[0040] Figure 2 is a flowchart illustrating an example of a series of processes from the generation of a first RAW image by the imaging device 100 to obtaining a developed image after development processing is complete. In the example shown in Figure 2, the imaging device 100 performs the process from shooting to generating the RAW image file, while the development application A, which operates on a separate device from the imaging device 100, performs the process from generating the developed image from the RAW image file. The imaging device 100 also generates an image file containing a second RAW image, which is generated by applying noise reduction processing to the first RAW image read from the image sensor.

[0041] Furthermore, Figure 3 is a flowchart showing the process shown in Figure 2, but with the noise reduction process performed by a development application B operating on a separate device from the imaging device 100.

[0042] In both Figure 2 and Figure 3, the development application can be run on a computer device such as a personal computer, tablet, or smartphone. The data transfer between devices that share the execution of the series of processes shown in Figures 2 and 3 may be performed via communication between the devices, or via external devices or recording media. Data transfer via external devices may involve, for example, uploading to and downloading from the external device. Data transfer via recording media may involve writing to and reading from recording media such as USB memory or memory cards.

[0043] In the flowcharts of Figures 2 and 3, the processing times for each device do not have to be consecutive. Furthermore, whether or not to apply noise reduction processing in the imaging device 100 may be specified by the user, for example, or the imaging device 100 may automatically determine this depending on the shooting mode and shooting conditions.

[0044] In this embodiment, the first device, which applies a portion of the image processing dependent on the imaging device that captured the RAW image to the RAW image, records information about the applied image processing in association with the RAW image. The second device, which applies the remaining portion of the image processing dependent on the imaging device to the RAW image to which a portion of the image processing dependent on the imaging device has been applied, sets the parameters of the image processing to be applied based on the information recorded in association with the RAW image. For example, the second device can change the parameters of the image processing to be applied or change the initial values ​​displayed in the user interface for adjusting the parameters of the image processing to be applied based on the information recorded in association with the RAW image.

[0045] For example, in the second case, where the second image processing is applied to the first device and the first image processing to the second device, the image quality after applying the image processing may be better than in the first case, where the first image processing is applied to the first device and the second image processing to the second device. This is the case, for example, when the intensity of the first image processing does not depend on the image height, but the intensity of the second image processing does.

[0046] Regarding the relationship between the first and second image processing steps, it is desirable from an image quality standpoint to lower the intensity of the second image processing step than the second image processing step in the first case. However, the user of the second device may not be aware of whether the first image processing step has been applied to the image to which the second image processing step is to be applied, or the intensity of the applied first image processing step. Furthermore, even if the user is aware of the intensity of the first image processing step, it is not easy to set an appropriate intensity for the second image processing step relative to the intensity of the first image processing step. Also, when the second device automatically performs the first image processing step, it is not possible to set the parameters of the first image processing step appropriately without information about the applied second image processing step.

[0047] Figure 4 is a block diagram showing an example of the functional configuration of an information processing device that can be used as a computer device to perform the development application shown in Figures 2 and 3. The information processing device may be any electronic device capable of executing the development application.

[0048] Display 401 is, for example, a liquid crystal display (LCD), which displays the screen provided by the program (OS and application program) being executed by the CPU 405.

[0049] The display controller 402 has video RAM (VRAM) and controls the display on the display 401.

[0050] The keyboard 403 and pointing device 404 are examples of user-operable input devices. The user can use the keyboard 403 and pointing device 404 to input characters and instructions into the information processing device, or to operate GUI elements such as icons and buttons on the screen displayed on the display 401.

[0051] The CPU 405 loads the program stored in the ROM 406 into the RAM 407 and executes it, thereby realizing the operation of the image processing system described below.

[0052] ROM406 stores programs executed by CPU405, various parameters, and settings. ROM406 may be electrically rewritable. RAM407 is used as a work area when CPU405 executes various programs, and as a buffer for various data.

[0053] The mass storage device 408 and the removable media drive (RMD) 409 are external storage devices. The mass storage device 408 may be a hard disk drive (HDD) or a solid-state drive (SSD), etc. The RMD 409 is a device that reads or writes to removable recording media such as a memory card.

[0054] Furthermore, the operating system, application programs (such as web browsers and photo editing applications), data, libraries, etc., executed by the information processing device are stored in one or more of the ROM 406 and mass storage devices 408, depending on their purpose.

[0055] Expansion slot 410 can accommodate various expansion boards, such as video capture boards and sound boards.

[0056] The network interface 411 is an interface for connecting the information processing device to a local network or an external network. The network interface 411 conforms to one or more wired and wireless communication standards.

[0057] Furthermore, in addition to the network interface 411, the information processing device may have communication interfaces with external devices that comply with various standards such as USB, HDMI (registered trademark), and Bluetooth (registered trademark).

[0058] Bus 412 consists of an address bus, a data bus, and a control bus, and connects the various functional blocks described above.

[0059] The following processes performed by the development application are implemented by the CPU 405 of the information processing device executing the development application stored in the ROM 406 or the mass storage device 408.

[0060] Returning to Figure 2, the imaging device 100 (system control unit 5) applies the first image processing in S201 to the first RAW image 150 read from the image sensor 2 to generate the second RAW image 160.

[0061] Here, the first image processing is assumed to be noise reduction processing, but other image processing methods whose intensity does not depend on the image height may also be used. Whether or not to apply the first image processing to the first RAW image 150, how many and what kind of image processing to apply as the first image processing, and the settings for the applied image processing will be either pre-configured or automatically determined by the system control unit 5 based on predetermined conditions.

[0062] For example, when applying noise reduction processing as the first image processing step, the system control unit 5 can automatically determine the intensity based on one or more of the shooting sensitivity, shooting mode, and shooting scenes determined by the image processing circuit 4. Specifically, the system control unit 5 can increase the intensity of the noise reduction processing in accordance with the increase in shooting sensitivity, or increase the intensity of the noise reduction processing when it is expected that the amount of noise in the captured image will increase. The same applies to the shooting mode and shooting scenes. Furthermore, the noise reduction processing may also be performed using a neural network such as an artificial neural network (ANN) or a convolutional neural network (CNN). Even with noise reduction processing using a neural network, the intensity of the noise reduction processing may be determined by taking into account the shooting sensitivity, shooting mode, and shooting scenes.

[0063] The system control unit 5 applies the first image processing to the RAW image using the image processing circuit 4 to generate a second RAW image. The image processing circuit 4 stores the generated second RAW image in the memory 3.

[0064] The system control unit 5 further generates additional information. This additional information is used when the imaging device 100 or a device other than the imaging device 100 applies image processing to the recorded second RAW image. The additional information may include, for example, the type of first image processing and the settings for its application (such as intensity and parameter values ​​corresponding to the image processing). The additional information may also include information that affects the image quality of the captured image, such as the noise characteristics of the image sensor and the characteristics (aberrations) of the imaging optical system.

[0065] The system control unit 5 associates the second RAW image 160 with additional information and records it, for example, on the recording medium 10. The second RAW image 160 and the additional information may be recorded in the same data file, or they may be recorded in separate data files. In either case, information such as shooting conditions is recorded separately from the additional information as EXIF ​​information in the data file that stores the second RAW image 160.

[0066] From S301 onward, the process is carried out by executing development application A on an information processing device 400 separate from the imaging device 100.

[0067] In S301, the CPU 405 acquires the second RAW image 160 to be processed and its associated additional information from a recording medium inserted in the RMD 409, or from an external device that can communicate via the network I / F 411, and stores them in the RAM 407.

[0068] Then, the CPU 405 determines the settings for the image processing (second image processing) to be applied to the second RAW image 160 based on the additional information. Specifically, first, the CPU 405 determines, based on the additional information, whether or not the second RAW image 160 has had a predetermined image processing applied to it.

[0069] The predetermined image processing may differ depending on the type of second image processing that the development application A intends to apply to the second RAW image. Therefore, this determination may be performed once the second image processing to be applied has been determined.

[0070] Here, we assume that in development application A, a second image processing step is applied, which has an intensity dependent on the image height. Therefore, the predetermined image processing step may be one that degrades image quality when the image processing step with an intensity dependent on the image height is applied. Here, as an example, let's assume that the predetermined image processing step is noise reduction. Performing noise reduction after the image processing step with an intensity dependent on the image height makes it easier to equalize the noise level in the image, which is advantageous in terms of image quality. Therefore, the predetermined image processing step can be said to be an image processing step that has an order in which the image quality changes depending on the execution order of the second image processing step.

[0071] If CPU405 does not determine that the second RAW image 160 has been subjected to predetermined image processing, it will set the settings for the second image processing to the default settings.

[0072] On the other hand, if CPU405 determines that the second RAW image 160 has been subjected to predetermined image processing, it changes the settings for the second image processing from the default settings. Specifically, CPU405 changes the settings for the second image processing to have less impact on image quality than the default settings.

[0073] For example, if the intensity of the first image processing step, noise reduction, is low, it is expected that the amount of noise in the second RAW image 160 will be greater than when the intensity is high. On the other hand, peripheral illumination correction is a process that multiplies the pixel value by a gain that increases as the image height increases, so the noise particularly in areas with high image height will increase significantly. Similarly, with optical aberration correction, the amount of correction increases with increasing image height, resulting in a bias in the noise distribution in the corrected image.

[0074] Therefore, based on the additional information, the CPU 405 determines or changes the setting so that the intensity of the second image processing is lower in order to suppress the increase in noise caused by the application of the second image processing. For example, the CPU 405 determines or changes the setting so that the lower the intensity of the noise reduction processing, the lower the intensity of the second image processing. If the development application A has a user interface that allows the user to adjust the intensity of the second image processing interactively, the CPU 405 can reflect the determined setting in the initial value of the user interface that adjusts the intensity. Alternatively, the CPU 405 may further determine an upper limit of the intensity based on the additional information and reflect the determined upper limit in the user interface that adjusts the intensity. In addition, the CPU 405 may notify the user via the user interface of the intensity of the second image processing at which the degradation of image quality becomes significant.

[0075] Furthermore, if the noise reduction processing intensity is high enough (for example, above the threshold) that the noise in the second RAW image is negligible, and the increase in noise due to the second image processing is negligible, then it is not necessary to change the intensity of the second image processing from the default setting.

[0076] Furthermore, the CPU 405 may consider the characteristics of the image sensor and imaging optical system included in the additional information when determining the settings for the second image processing. For example, the CPU 405 can estimate the amount of noise before applying noise reduction processing based on the sensitivity at the time of shooting and the noise characteristics of the image sensor. Therefore, the CPU 405 can more accurately estimate the amount of noise contained in the second RAW image based on the intensity of the noise reduction processing, and as a result, it can more accurately determine the settings for peripheral illumination correction processing. In optical aberration correction processing as well, the magnitude of aberrations in the peripheral parts of the image can be accurately estimated based on the characteristics of the imaging optical system and the aperture value at the time of shooting, and as a result, the settings for optical aberration correction processing can be more accurately determined.

[0077] In S302, CPU405 applies peripheral illumination correction processing to the second RAW image according to the settings determined in S301 (or settings further adjusted by the user).

[0078] In S303, CPU405 applies optical aberration correction processing according to the settings determined in S301 (or settings further adjusted by the user). Note that Figure 3 shows an example where peripheral illumination correction processing and optical aberration correction processing are applied in sequence, but either one or the other may be applied. Other image processing may also be applied.

[0079] For example, when a user instructs development application A to perform development processing, the CPU 405 generates a developed image according to the parameters set at that time and records it in the RMD 409 or mass storage device 408.

[0080] In Figure 3, the process is the same as in Figure 2, except that the processing S201 in Figure 2 is performed by development application B, which runs on another device capable of communicating with the imaging device 100, to generate a second RAW image. The device running development application B may be the same as the information processing device 400.

[0081] As described above, this embodiment provides an image processing apparatus and an image processing method that can assist in appropriately setting the parameters of image processing to be applied to image data to which image processing has been applied by another device.

[0082] (Other embodiments) In the embodiment described above, the development application determined appropriate settings for image processing to be applied to the second RAW image based on the additional information. However, the system control unit 5 of the imaging device 100 may perform a similar process, determining appropriate settings that take into account the first image processing for each image processing that may be applied to the second RAW image, and recording this as additional information.

[0083] In this case, the development application can obtain and use settings from the additional information that correspond to the image processing to be applied to the second RAW image.

[0084] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0085] This embodiment includes the following image processing apparatus, imaging apparatus, image processing method, and program. (Item 1) A means for acquiring RAW images and additional information, A determination means for determining whether or not the first image processing has been applied to the RAW image based on the additional information, A determination means for determining the settings of a second image processing to be applied to the RAW image according to the result of the determination, The processing means includes a means for applying the second image processing to the RAW image based on the settings determined by the determination means, If the determination means determines that the first image processing has been applied to the RAW image, it determines the settings for the second image processing so that the intensity of the second image processing is weaker than when it is not determined that the first image processing has been applied to the RAW image. An image processing apparatus characterized by the following: (Item 2) The image processing apparatus according to item 1, characterized in that the first image processing and the second image processing have a relationship such that applying the second image processing first results in better image quality after application. (Item 3) The image processing apparatus according to item 1 or 2, characterized in that the intensity of the second image processing depends on the image height, and the intensity of the first image processing does not depend on the image height. (Item 4) The image processing apparatus according to item 3, characterized in that the second image processing includes at least one of a process for correcting the effects of optical aberrations of a lens and a process for correcting vignetting. (Item 5) The image processing apparatus according to any one of items 1 to 4, characterized in that the first image processing is noise reduction processing. (Item 6) The additional information includes information on the intensity of the first image processing applied to the RAW image, The image processing apparatus according to any one of items 1 to 5, characterized in that the determination means determines the intensity of the second image processing in accordance with the intensity of the first image processing so as to suppress the increase in noise due to the application of the second image processing. (Item 7) The additional information further includes information regarding the noise characteristics of the image sensor, The image processing apparatus according to item 6, characterized in that the determination means determines the intensity of the first image processing and the intensity of the second image processing according to the noise characteristics so as to suppress the increase in noise due to the application of the second image processing. (Item 8) The first image processing is noise reduction processing, The image processing apparatus according to item 6 or 7, characterized in that the determination means determines the intensity of the second image processing in the same way as when it is determined that the first image processing is not applied to the RAW image, if the intensity of the first image processing is greater than or equal to a predetermined threshold. (Item 9) The determination means determines the initial value of the intensity of the second image processing as a setting for the second image processing, The image processing apparatus further includes control means for displaying a screen that includes a user interface for the user to set the intensity of the second image processing, The control means displays the user interface that reflects the initial value determined by the determination means. An image processing apparatus according to any one of items 1 to 8, characterized in that (Item 10) An acquisition means for acquiring a first image and additional information, A determination means for determining whether or not noise reduction processing has been applied to the first image based on the aforementioned additional information, A determination means that determines the setting of at least one of the processes for correcting the effect of optical aberrations of the lens applied to the first image and the process for correcting vignetting, in accordance with the result of the determination, The system includes a processing means that applies to the first image, based on the settings determined by the determination means, at least one of a process to correct the effect of optical aberrations of the lens and a process to correct peripheral vignetting. If the determination means determines that the noise reduction processing has been applied to the first image, it determines the setting of at least one of the processing to correct the effects of optical aberrations of the lens and the processing to correct peripheral vignetting such that the intensity of at least one of the processing to correct the effects of optical aberrations of the lens and the processing to correct peripheral vignetting are weaker than when it is determined that the noise reduction processing has not been applied to the first image. An image processing apparatus characterized by the following: (Item 11) A means for acquiring a first RAW image, Image processing means for generating a second RAW image by applying a first image processing to the first RAW image, A generation means for generating additional information used to apply a predetermined second image processing to the second RAW image, The system includes recording means for recording the second RAW image and the additional information in association, The imaging apparatus is characterized in that the generation means generates additional information corresponding to the intensity of the first image processing applied to the first RAW image. (Item 12) A means for acquiring a first image, Image processing means for generating a second image by applying noise reduction processing to the first image, A generation means for generating additional information used to apply to the second image at least one of a process for correcting the effects of optical aberrations of the lens and a process for correcting vignetting, The system includes recording means for recording the second image and the additional information in association, The imaging apparatus is characterized in that the generation means generates additional information corresponding to the intensity of the noise reduction process applied to the first image. (Item 13) An image processing method performed by an image processing device, To obtain RAW images and additional information, Based on the aforementioned additional information, it is determined whether or not the first image processing has been applied to the RAW image. Based on the result of the above determination, the settings for the second image processing to be applied to the RAW image are determined, The process involves applying the second image processing to the RAW image based on the determined settings, The determination described above includes determining the settings for the second image processing such that, if it is determined that the first image processing is applied to the RAW image, the intensity of the second image processing is weaker than if it is not determined that the first image processing is applied to the RAW image. An image processing method characterized by the following: (Item 14) To obtain the first image, Applying noise reduction processing to the first image to generate a second image, To generate additional information used to apply at least one of the following processes to the second image: a process to correct the effects of optical aberrations of the lens and a process to correct vignetting; The method includes recording the second image and the additional information in association with each other, An image processing method characterized by generating additional information corresponding to the intensity of the noise reduction processing applied to the first image. (Item 15) A program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 10.

[0086] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0087] 100: Imaging device, 4: Image processing circuit, 5: System control unit, 10: Recording medium, 13: Communication unit, 400: Information processing device, 405: CPU

Claims

1. An acquisition means for acquiring RAW images and additional information, A determination means for determining whether or not the first image processing has been applied to the RAW image based on the additional information, A determination means for determining the settings of a second image processing to be applied to the RAW image according to the result of the determination, The system includes a processing means that applies the second image processing to the RAW image based on the settings determined by the determination means, If the determination means determines that the first image processing is applied to the RAW image, it determines the settings for the second image processing such that the intensity of the second image processing is weaker than when it is not determined that the first image processing is applied to the RAW image. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, characterized in that the first image processing and the second image processing have a relationship such that applying the second image processing first results in better image quality after application.

3. The image processing apparatus according to claim 1, characterized in that the intensity of the second image processing depends on the image height, and the intensity of the first image processing does not depend on the image height.

4. The image processing apparatus according to claim 3, characterized in that the second image processing includes at least one of a process for correcting the effects of optical aberrations of the lens and a process for correcting vignetting.

5. The image processing apparatus according to claim 1, characterized in that the first image processing is noise reduction processing.

6. The additional information includes information on the intensity of the first image processing applied to the RAW image, The image processing apparatus according to claim 1, characterized in that the determination means determines the intensity of the second image processing in accordance with the intensity of the first image processing so as to suppress the increase in noise due to the application of the second image processing.

7. The additional information further includes information regarding the noise characteristics of the image sensor, The image processing apparatus according to claim 6, characterized in that the determination means determines the intensity of the first image processing and the intensity of the second image processing according to the noise characteristics so as to suppress the increase in noise due to the application of the second image processing.

8. The first image processing is noise reduction processing, The image processing apparatus according to claim 6, characterized in that the determination means determines the intensity of the second image processing in the same way as when it is determined that the first image processing is not applied to the RAW image, if the intensity of the first image processing is greater than or equal to a predetermined threshold.

9. The determination means determines the initial value of the intensity of the second image processing as a setting for the second image processing, The image processing apparatus further includes control means for displaying a screen that includes a user interface for the user to set the intensity of the second image processing, The control means displays the user interface that reflects the initial value determined by the determination means. The image processing apparatus according to feature 1.

10. An acquisition means for acquiring a first image and additional information, A determination means for determining whether or not noise reduction processing has been applied to the first image based on the additional information, A determination means that determines the setting of at least one of the processes for correcting the effect of optical aberrations of the lens applied to the first image and the process for correcting peripheral vignetting, in accordance with the result of the determination, The system includes a processing means that applies to the first image, based on the settings determined by the determination means, at least one of a process to correct the effect of optical aberrations of the lens and a process to correct peripheral vignetting. If the determination means determines that the noise reduction processing has been applied to the first image, it determines the setting of at least one of the processing for correcting the effects of optical aberrations of the lens and the processing for correcting peripheral vignetting such that the intensity of at least one of the processing for correcting the effects of optical aberrations of the lens and the processing for correcting peripheral vignetting are weaker than when it is determined that the noise reduction processing has been applied to the first image. An image processing apparatus characterized by the following:

11. A means for acquiring a first RAW image, Image processing means for generating a second RAW image by applying a first image processing to the first RAW image, A generation means for generating additional information used to apply a predetermined second image processing to the second RAW image, The system includes recording means for recording the second RAW image and the additional information in association, The imaging apparatus is characterized in that the generation means generates additional information corresponding to the intensity of the first image processing applied to the first RAW image.

12. A means for acquiring a first image, Image processing means for generating a second image by applying noise reduction processing to the first image, A generation means for generating additional information used to apply at least one of the following processes to the second image: a process for correcting the effects of optical aberrations of the lens and a process for correcting vignetting; The system includes recording means for recording the second image and the additional information in association, The imaging apparatus is characterized in that the generation means generates additional information corresponding to the intensity of the noise reduction process applied to the first image.

13. An image processing method performed by an image processing device, To acquire RAW images and additional information, Based on the aforementioned additional information, it is determined whether or not the first image processing has been applied to the RAW image. Based on the result of the above determination, the settings for the second image processing to be applied to the RAW image are determined, The process involves applying the second image processing to the RAW image based on the determined settings, The above determination includes determining the settings for the second image processing such that, if it is determined that the first image processing is applied to the RAW image, the intensity of the second image processing is weaker than when it is not determined that the first image processing is applied to the RAW image. An image processing method characterized by the following:

14. To obtain the first image, Applying noise reduction processing to the first image to generate a second image, To generate additional information used to apply at least one of the following processes to the second image: a process to correct the effects of optical aberrations of the lens and a process to correct vignetting; The method includes recording the second image and the additional information in association with each other, An image processing method characterized by generating additional information corresponding to the intensity of the noise reduction processing applied to the first image.

15. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 10.

Images