Image processing device, control method for image processing device, and program

Abstract

This allows for the recording of pre-development image data, intended for high-quality editing and image processing after shooting, while minimizing strain on storage capacity. [Solution] Pre-development image data and post-development image data obtained by applying image processing to the pre-development image data are acquired from the buffer unit, and image quality defects are detected from the post-development image data. If an area with image quality defects is detected, the area in which the image quality defects were detected is cut out from the pre-development image data to generate partial pre-development image data. The generated pre-development image data is then associated with the compressed post-development image data and recorded on the recording medium.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for the image processing apparatus, and a program.

Background Art

[0002] In an imaging device such as a digital camera, an image passing through an optical lens is converted into an image signal by an imaging element such as a CCD sensor or a CMOS sensor, and the image signal is A / D converted to generate RAW data (pre-developed image data). Thereafter, based on imaging conditions such as development processing, white balance, sharpness and contrast adjustment, noise removal processing, aberration correction processing, etc. for the pre-developed image data, post-developed image data that can be visually recognized on the monitor of each device is generated. Further, the post-developed image data is compressed, and image data in a data-compressed image format such as JPEG data or HEIF data (post-compression image data) is output.

[0003] Since the post-compression image data is compressed from the post-developed image data and has a smaller data volume, a large number of post-compression image data can be recorded on a recording medium. However, since the post-compression image data has been developed and the original light information cannot be fully restored, the range in which the image quality can be adjusted is narrow, and a situation where it is difficult to recover poor image quality occurs.

[0004] On the other hand, the pre-developed image data is data that directly records the light information captured by the imaging element, has a deep bit depth, and high-quality images can be generated by various image processes from the pre-developed image data. Also, by recording the pre-developed image data, a large-scale NN (Neural Network) model that was not applied in the imaging device at the time of imaging can be applied, and further improvement in image quality can be expected. However, the pre-developed image data has a large data volume, and there is a concern that continuous shooting and recording will strain the capacity of the recording medium. For these reasons, there is a desire to efficiently record only the necessary pre-developed image data so that the desired scene can be improved in image quality.

[0005] Therefore, Patent Document 1 discloses a method for saving recording medium capacity by deciding whether or not to record RAW data based on an evaluation of the image quality characteristics in the captured image. Furthermore, Patent Document 2 discloses a method for outputting RAW data with a small recording capacity by automatically determining the cropping area. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-126853 [Patent Document 2] Japanese Patent Publication No. 2006-33291 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the method described in Patent Document 1 records the entire RAW data when recording RAW data, so it is not possible to sufficiently reduce the recording capacity. Also, the method described in Patent Document 2 records only a portion of the RAW data of the main subject area, so if there is image quality defect outside the main subject area, it is not possible to recover from the image quality defect using the RAW data.

[0008] In view of the aforementioned problems, the present invention aims to enable the recording of pre-development image data, intended for high-quality image enhancement through post-shooting editing and image processing, while minimizing strain on recording capacity. [Means for solving the problem]

[0009] The image processing apparatus according to the present invention is characterized by comprising: acquisition means for acquiring first image data and second image data obtained by performing image processing on the first image data; detection means for detecting areas of poor image quality from the second image data; generation means for generating third image data including the areas of poor image quality detected by cutting out from the first image data; and recording means for recording the third image data on a recording medium in association with the second image data. [Effects of the Invention]

[0010] According to the present invention, pre-development image data, intended for high-quality image enhancement through post-shooting editing and image processing, can be recorded while minimizing strain on recording capacity. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of the internal configuration of an image processing apparatus according to an embodiment. [Figure 2] This flowchart shows an example of a processing procedure performed by the image processing device in the first embodiment. [Figure 3] This is a diagram illustrating a method for detecting areas with poor image quality. [Figure 4] This diagram illustrates the file names and data structure. [Figure 5] This flowchart shows an example of a processing procedure performed by the image processing device in the second embodiment. [Figure 6] This diagram illustrates the details of the related information in the image data before partial development. [Figure 7] This flowchart shows an example of a processing procedure performed by the image processing device in the fourth embodiment. [Figure 8] This flowchart shows an example of a processing procedure performed by the image processing device in the fifth embodiment. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below, and various forms that do not depart from the spirit of the invention are also included. Furthermore, each embodiment described below is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0013] (First embodiment) Figure 1 is a block diagram showing an example of the internal configuration of the image processing apparatus 100 according to this embodiment. As shown in Figure 1, the image processing apparatus 100 comprises a control unit 101, a storage unit 102, a recording medium 103, a buffer unit 104, an operation unit 105, a display unit 106, an optical system 111, an image sensor 112, and an A / D converter 113. Furthermore, the image processing apparatus 100 comprises an image processing unit 114, a data acquisition unit 115, an image quality defect detection unit 116, a partial image generation unit 117, a recording unit 118, a status confirmation unit 119, a priority setting unit 120, and a data deletion unit 121. These components are connected via a system bus 130.

[0014] The control unit 101 is a control unit in which the CPU (Central Processing Unit) functions, performing calculations and logical decisions for various processes, and controlling each component connected to the system bus 130. The memory unit 102 is a memory such as ROM (Read-Only Memory) or RAM (Random Access Memory). The ROM stores the control program executed by the control unit 101. The RAM is used as a workspace when the control unit 101 executes the control program. Alternatively, program memory can be realized by loading a program into the RAM from an external storage device connected to the image processing device 100.

[0015] The recording medium 103 is a memory card such as an SD card that can be attached and removed, or a built-in hard disk, and various types of data are recorded on it. The buffer unit 104 is a memory that temporarily stores pre-development image data such as RAW data, post-development image data, and compressed image data. The operation unit 105 is an operation member such as a physical button or dial, and various shooting-related settings can be input. The display unit 106 is composed of a liquid crystal display or the like, and displays information such as the captured image, the setting screen, and the processing status to the user. Further, the display unit 106 may include a touch panel. In this case, it has the same function as the operation unit 105, and various settings can be input according to the display content.

[0016] The optical system 111 is composed of a lens or the like, and is a configuration for passing external light so as to form an optical image of the shooting object on the image sensor 112. The image sensor 112 is a CCD sensor, a CMOS sensor, or the like, and converts the light incident from the optical system 111 into an electrical signal. The A / D converter 113 converts the analog electrical signal into a digital image and generates pre-development image data (RAW data).

[0017] The image processing unit 114 performs image processing such as improving image quality, sharpening, and color adjustment on the pre-development image data to generate new image data, or performs development processing based on the imaging conditions of the pre-development image data and converts it into post-development image data. Further, the image processing unit 114 performs compression processing on the post-development image data to generate compressed image data in JPEG format, TIFF format, or the like.

[0018] The data acquisition unit 115 reads and acquires pre-development image data, post-development image data, or compressed data from the recording medium 103 or the buffer unit 104. The image quality defect detection unit 116 detects image quality defects such as insufficient noise removal, artifacts, white spots, and black crush from the pre-development image data, post-development image data, or compressed data acquired by the data acquisition unit 115. The partial image generation unit 117 cuts out the pre-development image data in a spatial region and generates partial pre-development image data. The recording unit 118 records the pre-development image data and information related to the pre-development image data, along with the corresponding compressed image data, onto the recording medium 103.

[0019] The status confirmation unit 119 checks the available capacity of the recording medium 103. The priority setting unit 120 sets the recording priority of the pre-developed partial image data using the type and degree of image quality defects detected by the image quality defect detection unit 116, the size of the pre-developed partial image data extracted by the partial image generation unit 117, and other factors. The data deletion unit 121 deletes various types of data from the recording medium 103.

[0020] In this embodiment, the image processing device 100 is described as an imaging device such as a digital camera. In this embodiment, noise reduction processing using an NN model is applied to the pre-development image data, and development processing is performed. However, the generation and recording of captured images of a partial region where noise remains in the post-development image data or the compressed image data will be described.

[0021] Figure 2 is a flowchart showing an example of the processing procedure performed by the image processing device 100 in this embodiment. The processing shown in Figure 2 is started when the device receives a shooting instruction from the operation unit 105.

[0022] First, in step S201, image capture is performed under the control of the control unit 101. Specifically, light that passes through the optical system 111 and enters the image sensor 112 is converted into an analog signal by the image sensor 112. Then, the A / D converter 113 converts the analog signal into a digital image (pre-development image data) and stores it in the buffer unit 104.

[0023] Next, in step S202, the image processing unit 114 processes the pre-development image data stored in the buffer unit 104 to generate post-development image data. The image processing unit 114 then stores the generated post-development image data together with the pre-development image data in the buffer unit 104. As mentioned above, the development process includes multiple processes such as debayering, white balance, sharpness and contrast adjustment, noise reduction, and aberration correction. The image processing unit 114 also compresses the post-development image data to generate compressed image data such as JPEG data. The recording unit 118 then records the generated compressed image data onto the recording medium 103.

[0024] Next, in step S203, the data acquisition unit 115 acquires a pair of pre-development image data and post-development image data from the buffer unit 104. Alternatively, a pair of pre-development image data and compressed image data may be acquired. The following explanation will use the case where a pair of pre-development image data and post-development image data is acquired as an example.

[0025] Next, in step S204, the image quality defect detection unit 116 detects areas in the developed image data where image quality defects exist. The image quality defects detected in this embodiment are due to insufficient noise reduction during the noise reduction process, and the areas where image quality defects occur are identified using spatial frequencies and pixel values. More specifically, the image quality defect detection unit 116 generates segmented images by spatially dividing the developed image data, and obtains frequency spectrum information by performing a Fourier transform on each segmented image. Then, if the proportion of intensity in the portion of the acquired frequency spectrum that is higher than a specific frequency is greater than a specific value, the image quality defect detection unit 116 considers that there is a large amount of remaining noise and determines that image quality defects exist in that segmented area.

[0026] Figure 3 illustrates a method for detecting areas of poor image quality. For example, to detect noise present on the subject 301 in the developed image data 300 shown in Figure 3 as poor image quality, divided images are generated by spatially dividing the developed image data 310 with dashed lines. Then, if the proportion of high-frequency component intensity in the frequency spectrum of each divided image is greater than a predetermined value, it is determined that poor image quality exists. Note that the method for detecting areas where noise remains is not limited to the above, and other methods may be used. In this embodiment, the proportion of high-frequency component intensity was calculated by dividing each region into rectangles as in the developed image data 310, but the frequency spectrum may also be obtained for each object using object detection.

[0027] Next, in step S205, the image quality defect detection unit 116 determines whether or not image quality defects have been detected in the developed image data. If image quality defects are detected as a result of this determination, the process proceeds to step S206. On the other hand, if no image quality defects are detected, the subsequent processing is skipped, and the process shown in Figure 2 is terminated.

[0028] In step S206, the partial image generation unit 117 extracts a rectangular region from the pre-development image data that includes the area where the image quality defect is determined to exist by the image quality defect detection unit 116, and generates partial pre-development image data. If multiple image quality defects are detected, multiple partial pre-development image data are generated. If the divided images where image quality defects are determined to exist are adjacent to each other, they may be extracted together as a single image, as shown in the partial pre-development image data 320 in Figure 3, rather than being extracted individually. The shape of the extraction is not limited to a rectangle; it may also be extracted along the outline of an object.

[0029] Next, in step S207, the recording unit 118 records the partially undeveloped image data generated by the partially image generation unit 117 onto the recording medium 103. At this time, the file name of the partially undeveloped image data to be recorded is referenced to the file name of the corresponding compressed image data to establish an association between the two.

[0030] Figure 4 is a diagram illustrating file names and data structures. Here, as shown in Figure 4(a), compressed image data and pre-developed image data are recorded in a specific storage area 400 of the recording medium 103. As shown in combination 410, if the file name of the corresponding compressed image data 411 is "filename1.(compressed image data extension)", the file name of the pre-developed image data 412 is set to "filename1.(pre-developed image data extension)" and recorded. Similarly, as shown in combinations 420 and 430, compressed image data 421 and 431 and pre-developed image data 422 and 432 are associated with the corresponding file names and recorded.

[0031] Furthermore, if multiple pre-development image data are generated for a compressed image data, each pre-development image data should be distinguishable. As shown in combination 440 in Figure 4(b), numbers or other identifiers may be added for recording. For example, when associating multiple pre-development image data 442-444 with a compressed image data 411, the file names should be recorded as "filename1_1.(pre-development image data extension)" to "filename1_M.(pre-development image data extension)" respectively.

[0032] Alternatively, to associate and record the pre-development image data and the compressed image data, one can set the file names of both to any name and record both files in the same dedicated folder. In this case, one folder is created for each developed or compressed image data that generated the pre-development image data.

[0033] By doing this, if there are noisy areas in the developed image data due to insufficient noise reduction, it becomes possible to perform noise reduction processing on the recorded partial pre-development image data using editing software or similar tools, utilizing a large-scale NN model. By combining the noise-reduced and developed partial post-development image data with the associated compressed image data, a single high-quality image can be obtained. Furthermore, since the recorded partial pre-development image data is a partial image, its data size is reduced compared to the pre-development image data, thus avoiding strain on the recording capacity of the storage medium.

[0034] As described above, according to this embodiment, pre-development image data of only the portion containing image quality defects is recorded in association with the compressed image data. This makes it possible to obtain high-quality images through image processing while avoiding strain on the recording capacity of the recording medium.

[0035] <Variation 1-1> In this embodiment, insufficient noise reduction was described as an example of poor image quality. However, the poor image quality detection unit 116 may also detect blown-out highlights or crushed blacks in the developed image data or compressed image data as examples of poor image quality. In this case, the poor image quality detection unit 116 checks the pixel values ​​of pixels in the developed image data or compressed image data, and determines that an area where pixels with a specific pixel value or higher are clustered is larger than a specific size, as a blown-out highlight area. Similarly, it determines that an area where pixels with a specific pixel value or lower are clustered is larger than a specific size, as a crushed black area. When detecting blown-out highlight areas or crushed black areas, the poor image quality detection unit 116 checks the pixel values ​​of the corresponding areas in the pre-developed image data. If the pixel values ​​in the corresponding areas in the pre-developed image data are saturated or at the minimum possible value, it is difficult to recover the blown-out highlights or crushed blacks even if partial pre-developed image data is generated. For this reason, in such cases, the processing in steps S206 and S207 may be skipped, and the flow shown in Figure 2 may be terminated.

[0036] <Variation 1-2> Furthermore, the image quality defect detection unit 116 may also detect artifacts, which are depictions that do not exist in the real world, as image quality defects. In this case, the image quality defect detection unit 116 scans for shapes that appear as artifacts using a shape scanning method such as template matching, and determines that an area is an artifact region if the corresponding shape exists.

[0037] <Variation 1-3> Furthermore, the image quality defect detection unit 116 may detect multiple types of image quality defects in one developed image file or one compressed image file. For example, the image quality defect detection unit 116 may simultaneously detect insufficient noise reduction and artifacts, and record the detected areas with insufficient noise reduction and the areas containing artifacts as partially pre-developed image data, respectively.

[0038] (Second embodiment) A second embodiment of the present invention will be described below. The internal configuration of the image processing apparatus according to this embodiment is the same as that shown in Figure 1, so its description will be omitted. Only the differences from the first embodiment will be described below. In this embodiment, the convenience of image editing is enhanced by recording information regarding the compressed image data and the image data before partial development. Similar to the first embodiment, the image quality defects described in this embodiment are those where the noise reduction effect of the noise reduction process in the digital camera is weak, resulting in residual noise regions in the developed image data or the compressed image data.

[0039] Figure 5 is a flowchart showing an example of the processing procedure performed by the image processing device 100 in this embodiment. Steps S201 to S203 in Figure 5 are the same as steps S201 to S203 in Figure 2, so their explanation is omitted.

[0040] In step S501, as described in step S204 above, the image quality defect detection unit 116 detects areas in the developed image data where image quality defects exist. In this embodiment, the image quality defect detection unit 116 further stores information on the type of image quality defect detected (hereinafter referred to as image quality defect type information) and information on the degree of the image quality defect in the buffer unit 104. Here, the image quality defect type information is information on the type of image quality defect, such as insufficient noise reduction, artifacts, blown-out highlights, and crushed blacks. The information on the degree of image quality defect represents the degree of each image quality defect, with a larger value indicating a worse degree of image quality defect. In the case of insufficient noise reduction, if the proportion of high-frequency intensity is high, it may be expressed as a qualitative level such as "High" or "High," or as a quantitative score. The next step S205 is the same as step S205 in Figure 2, so the explanation is omitted.

[0041] In step S502, the partial image generation unit 117 extracts a rectangular region from the pre-development image data that includes the region where image quality defects were determined to exist in step S501, and generates partial pre-development image data. In this embodiment, the partial image generation unit 117 further records the position information of the partial pre-development image data on the recording medium 103. The position information of the partial pre-development image data is information for determining which position within the compressed image data was extracted. Specifically, it may be represented by a pair of "top-left position coordinates" and "bottom-right position coordinates," or a pair of "top-left position coordinates," "region width," and "region height." The next step S207 is the same as step S207 in Figure 2, so the explanation is omitted.

[0042] In step S503, the recording unit 118 records the name of the pre-development image data, along with the type of image quality defect, the degree of image quality defect, and the position information of the pre-development image data, in the metadata of the compressed image data. For example, as shown in Figure 6(a), the metadata area 600 of the compressed image data records related information 610 to 630 of the pre-development image data related to the compressed image data. At this time, one set of related information is associated with each pre-development image data.

[0043] Figure 6(b) also shows an example of related information 640 for pre-developed image data. Related information 640 includes the pre-developed image name 641, the X-coordinate start position 642, the Y-coordinate start position 643, the X-coordinate end position 644, the Y-coordinate end position 645, the type of image quality defect information 646, and the degree of image quality defect 647. In this embodiment, an example including seven types of information as related information is shown, but it is not necessary to include all seven types, and other types of information may also be included. However, at least one of the following information should be included: the type of image quality defect information, the degree of image quality defect information, and the position information of the pre-developed image data.

[0044] According to this embodiment, when a user adjusts the image quality of pre-processed image data using image editing software, they can utilize the image quality defect type information and the degree of image quality defect information stored in the metadata. This makes it possible to automatically set image processing parameters and the applied NN model, thereby reducing the user's effort. Furthermore, by utilizing the position information of the pre-processed image data, when the pre-processed image data that has undergone image quality adjustment and processing is combined into the compressed image data, it becomes possible to automatically adjust the position and combine it in the correct location.

[0045] <Modification 2> In this embodiment, an example of recording each piece of information in the metadata of the compressed image data has been described, but this information may also be recorded in the metadata of each pre-developed image data. Specifically, for each pre-developed image data, the metadata may include information on the type of image quality defect, information on the degree of image quality defect, location information of the pre-developed image data, and the name of the corresponding compressed image data.

[0046] Furthermore, as shown in Figure 6(c), this information may be recorded in a different management file 650 from the compressed image data and the pre-developed image data. In this case, for example, the management file 650 manages this information using a CSV or XML file. As shown in Figure 6(c), the management file 650 includes pre-developed image data 1 related information 651 to pre-developed image data L related information 653. This related information is then used to create the pre-developed image data related information 660 as shown in Figure 6(d). Specifically, it includes the pre-developed image name 661, the X-coordinate start position 662, the Y-coordinate start position 663, the X-coordinate end position 664, the Y-coordinate end position 665, image quality defect type information 666, image quality defect degree 667, and the compressed image name 668.

[0047] (Third embodiment) A third embodiment of the present invention will be described below. The internal configuration of the image processing apparatus according to this embodiment is the same as that in Figure 1, so the explanation will be omitted. Also, the processing procedure by the image processing apparatus in this embodiment is basically the same as that in Figure 2, so the explanation will be omitted. Below, only the differences from the first embodiment will be described. In the first and second embodiments, when multiple areas with image quality defects were detected, multiple pre-developed partial image data were generated. In contrast, in this embodiment, multiple areas with image quality defects are detected, and if predetermined conditions are met, a single pre-developed partial image data is generated that encompasses multiple areas with image quality defects. Hereinafter, image quality defects will be described as those in which the noise reduction effect of the noise reduction process in the digital camera is weak, and noise areas remain in the developed image data or compressed image data.

[0048] In step S206, if multiple areas with poor image quality are detected in step S204, the partial image generation unit 117 calculates the distance between these areas within the developed image data. If the calculated distance between the areas with poor image quality is smaller than a predetermined value, the unit combines and cuts out these areas to encompass them, generating a single partial image data file before development.

[0049] As described above, according to this embodiment, when multiple areas with image quality defects are detected, areas that are close together are generated as a single pre-developed partial image data. This reduces the number of pre-developed partial image data generated compared to the first embodiment, thereby reducing the user's effort when editing or correcting images. In this embodiment, the decision of whether or not to combine areas is made based on the distance between areas in the developed image data or the compressed image data, but areas with the same type of image quality defect may be combined. Alternatively, areas with the same type of image quality defect and whose degree of defect is close within a predetermined range may be combined.

[0050] (Fourth embodiment) A fourth embodiment of the present invention will be described below. The internal configuration of the image processing apparatus according to this embodiment is the same as that shown in Figure 1, so its description will be omitted. Only the differences from the first embodiment will be described below. In this embodiment, a recording priority is set for each pre-developed image data, and the pre-developed image data to be recorded is changed according to the available capacity of the recording medium 103. In addition, in this embodiment, when the available capacity of the recording medium 103 is low, the pre-developed image data in the recording medium 103 is deleted according to the recording priority of each pre-developed image data, and new pre-developed image data is recorded.

[0051] Figure 7 is a flowchart showing an example of the processing procedure performed by the image processing device 100 in this embodiment. Steps S201 to S203 in Figure 7 are the same as steps S201 to S203 in Figure 2, so their explanation is omitted.

[0052] In step S701, the status confirmation unit 119 confirms the recordable capacity of the recording medium 103. Then, in step S702, the image quality defect detection unit 116 adjusts the thresholds that serve as criteria for determining various image quality defects, according to the recordable capacity of the recording medium 103 confirmed in step S701. When the recordable capacity of the recording medium 103 is large, the thresholds are relaxed so that even images with a low degree of image quality defect are judged as defective. Conversely, when the recordable capacity of the recording medium 103 is small, the thresholds are tightened so that only images with a high degree of image quality defect are judged as defective. For example, when detecting image quality defects due to insufficient noise reduction, the threshold for the ratio of high-frequency intensity is adjusted. In this way, when the recordable capacity of the recording medium is small, only the necessary pre-development image data can be generated, thereby preventing strain on the recordable capacity. The next steps S204 and S205 are the same as steps S204 and S205 in Figure 2, respectively, so their explanation is omitted.

[0053] In step S703, the partial image generation unit 117 adjusts the range from which to extract the partial pre-development image data to the pre-development image data, according to the recordable capacity of the recording medium 103 confirmed in step S701. Then, it generates the partial pre-development image data according to the adjusted range. When the recordable capacity of the recording medium 103 is large, it extracts the image with a larger margin to include the area with image quality defects and the surrounding area. On the other hand, when the recordable capacity of the recording medium 103 is small, it extracts the image without leaving any margin around the area with image quality defects. This allows more pre-development image data to be recorded on the recording medium.

[0054] In step S704, the priority setting unit 120 sets a recording priority for each of the pre-developed image data generated in step S703. The recording priority is used as a criterion for selecting pre-developed image data to delete when the recordable capacity of the recording medium 103 runs out. The recording priority is set based on at least one piece of information from the type of image quality defect, the degree of image quality defect, and the position of the pre-developed image data. For example, pre-developed image data with a low degree of image quality defect is set to have a lower recording priority. In addition, other information may be used, and the recording priority may be set in combination using multiple pieces of information.

[0055] In step S705, the status confirmation unit 119 checks whether the pre-developed partial image data generated in step S703 can be recorded on the recording medium 103. If, as a result of this determination, there is sufficient recording capacity on the recording medium 103 and the generated pre-developed partial image data can be recorded, the process proceeds to step S207. On the other hand, if there is insufficient recording capacity on the recording medium 103 and the generated pre-developed partial image data cannot be recorded, the process proceeds to step S706.

[0056] In step S706, the data deletion unit 121 deletes pre-developed image data from the recording medium 103 in order to secure recordable capacity for recording the pre-developed image data generated in step S703. At this time, pre-developed image data with the lowest recording priority set in step S704 is deleted first. Pre-developed image data from the recording medium 103 is deleted until recordable capacity for recording the pre-developed image data generated in step S703 is secured in the recording medium 103. The next step S207 is the same as step S207 in Figure 2, so the explanation is omitted. However, when recording pre-developed image data, the recording unit 118 records the recording priority information in association with the pre-developed image data. The recording priority information may be stored in the metadata of the pre-developed image data, or it may be managed in a different management file from the pre-developed image data.

[0057] As described above, this embodiment records pre-development image data based on a variable criterion that takes into account the available storage capacity. This allows users to take pictures without worrying about the available storage capacity. Furthermore, when the available storage capacity runs out, pre-development image data is deleted by referring to the recording priority, so that pre-development image data that is more likely to be edited is recorded first.

[0058] (Fifth embodiment) The fifth embodiment of the present invention will be described below. The internal configuration of the image processing apparatus according to this embodiment is the same as that shown in Figure 1, so its description will be omitted. Only the differences from the first embodiment will be described below. In the first to fourth embodiments, an imaging apparatus equipped with an imaging unit was described as an example of an image processing apparatus. In this embodiment, however, an example of reducing the recording capacity of the recording medium 103 of an information processing apparatus that does not have an imaging unit will be described. In this embodiment, it is assumed that the recording medium 103 has a set of pre-development image data and compressed image data for the entire area of ​​the image pre-recorded. Then, partial pre-development image data including the area where image quality defects exist is generated from one or more pre-development image data, and the capacity of the recording medium 103 is reduced by deleting the original pre-development image data.

[0059] Figure 8 is a flowchart showing an example of the processing procedure performed by the image processing device 100 in this embodiment. In this embodiment, the processing shown in Figure 8 is started when the device receives an instruction from the operation unit 105 to generate pre-developed image data. First, in step S801, the data acquisition unit 115 acquires a pair of pre-development image data and compressed image data generated based on the pre-development image data from the recording medium 103. The next steps S204 to S207 are the same as steps S204 to S207 in Figure 2, so their explanation is omitted. However, in this embodiment, areas where image quality defects exist are detected from the compressed image data. If no image quality defects are detected as a result of the determination in step S205, the process proceeds to step S802.

[0060] In step S802, the data deletion unit 121 deletes the pre-development image data acquired in step S801 from the recording medium 103. If image quality defects are detected in the compressed image data, partial pre-development image data is generated in step S206 and recorded in step S207. Therefore, the recordable capacity can be increased by deleting the original pre-development image data. Also, if no image quality defects are detected in the compressed image data, it can be assumed that the image quality of the compressed image data is sufficient, and therefore there is no problem in deleting the pre-development image data. Thus, the recordable capacity can also be increased in such cases.

[0061] As described above, according to this embodiment, even when pre-development image data is recorded on the recording medium, the recordable capacity of the recording medium can be increased.

[0062] (Other embodiments) In the fourth embodiment described above, in step S702, the criteria for determining poor image quality were changed according to the recordable capacity of the recording medium, and in step S703, the range to be cropped was adjusted according to the recordable capacity of the recording medium. On the other hand, the processes in steps S702 and S703 may also be applied in the first to third and fifth embodiments, and either the process in step S702 or step S703 may be applied in the first to third and fifth embodiments.

[0063] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0064] This embodiment includes the following configurations, methods, and programs.

[0065] (Composition 1) Acquisition means for acquiring first image data and second image data obtained by applying image processing to the first image data, A detection means for detecting areas of poor image quality from the second image data, A generation means for generating a third image data that includes the region where the image quality defect was detected, by cutting out the first image data, A recording means for recording the third image data in association with the second image data onto a recording medium, An image processing apparatus characterized by comprising:

[0066] (Configuration 2) The image processing apparatus according to configuration 1, characterized in that the recording means records at least one piece of information from the position from which the third image data was extracted, the type of image quality defect, and the degree of the image quality defect when recording the third image data. (Composition 3) The image processing apparatus according to configuration 2, characterized in that the recording means records at least one piece of information from the position where the third image data was extracted, the type of image quality defect, and the degree of the image quality defect in the metadata of the second image data, the metadata of the third image data, or in a management file different from the second image data and the third image data.

[0067] (Composition 4) The image processing apparatus according to any one of configurations 1 to 3, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data so as to encompass the multiple regions. (Composition 5) The image processing apparatus according to any one of configurations 1 to 4, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data so as to encompass the regions among the multiple regions where the degree of the image quality defect falls within a predetermined range. (Composition 6) The image processing apparatus according to any one of configurations 1 to 5, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data such that the multiple regions include regions where the distance between the regions is less than a predetermined value.

[0068] (Composition 7) A setting means for setting a priority for recording the third image data on the recording medium based on at least one piece of information from which the third image data is extracted, the type of image quality defect, and the degree of image quality defect; A deletion means that deletes the third image data from the recording medium based on the priority set by the setting means, An image processing apparatus according to any one of configurations 1 to 6, further comprising the above. (Composition 8) If the recording capacity of the recording medium is insufficient to record the third image data generated by the generation means using the recording means, the deletion means deletes the third image data recorded on the recording medium in order of lowest priority as set by the setting means. The image processing apparatus according to configuration 7, characterized in that the recording means records the third image data generated by the generation means onto the recording medium after the recording capacity for recording has been secured by deletion by the deletion means.

[0069] (Composition 9) The image processing apparatus according to any one of configurations 1 to 8, characterized in that the generation means generates the third image data while adjusting the range to be cut out from the first image data according to the available capacity of the recording medium. (Composition 10) The image processing apparatus according to configuration 9, characterized in that the generation means generates the third image data so as to include the region where the image quality defect is detected and the region surrounding the region, and adjusts the size of the surrounding region according to the available capacity of the recording medium. (Composition 11) The image processing apparatus according to any one of configurations 1 to 10, characterized in that the detection means changes the criteria for detecting poor image quality according to the available capacity of the recording medium.

[0070] (Composition 12) An image processing apparatus according to any one of configurations 1 to 11, further comprising a deletion means for deleting the first image data from the recording medium when the third image data is recorded on the recording medium by the recording means, provided that the first image data is recorded on the recording medium. (Composition 13) The image processing apparatus according to any one of configurations 1 to 12, characterized in that the detection means detects at least one of noise, black crushing, white clipping, and artifacts as the image quality defect.

[0071] (method) An acquisition step of acquiring first image data and second image data obtained by applying image processing to the first image data, A detection step for detecting areas of poor image quality from the second image data, A generation step of generating a third image data that includes the region where the image quality defect was detected by cutting out the first image data, A recording step of recording the third image data in association with the second image data onto a recording medium, A control method for an image processing apparatus, characterized by comprising:

[0072] (program) An acquisition step of acquiring first image data and second image data obtained by applying image processing to the first image data, A detection step for detecting areas of poor image quality from the second image data, A generation step of generating a third image data that includes the region where the image quality defect was detected by cutting out the first image data, A recording step of recording the third image data in association with the second image data onto a recording medium, A program that causes a computer to execute something. [Explanation of symbols]

[0073] 101: Control unit, 103: Recording medium, 115: Data acquisition unit, 116: Image quality defect detection unit, 117: Partial image generation unit, 118: Recording unit

Claims

1. Acquisition means for acquiring first image data and second image data obtained by performing image processing on the first image data, A detection means for detecting areas of poor image quality from the second image data, A generation means for generating a third image data that includes the region where the image quality defect was detected, by cutting out the first image data, A recording means for recording the third image data in association with the second image data onto a recording medium, An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the recording means records at least one piece of information from the position from which the third image data was extracted, the type of image quality defect, and the degree of the image quality defect when recording the third image data.

3. The image processing apparatus according to claim 2, characterized in that the recording means records at least one piece of information from the position where the third image data was extracted, the type of image quality defect, and the degree of the image quality defect in the metadata of the second image data, the metadata of the third image data, or in a management file different from the second image data and the third image data.

4. The image processing apparatus according to claim 1, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data so as to encompass the multiple regions.

5. The image processing apparatus according to claim 1, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data such that the regions among the multiple regions have a degree of image quality defect within a predetermined range.

6. The image processing apparatus according to claim 1, characterized in that when the detection means detects the same type of image quality defect from multiple regions, the generation means generates the third image data such that the multiple regions include regions where the distance between the regions is less than a predetermined value.

7. A setting means for setting a priority for recording the third image data on the recording medium based on at least one piece of information from which the third image data is extracted, the type of image quality defect, and the degree of image quality defect, A deletion means that deletes the third image data from the recording medium based on the priority set by the setting means, The image processing apparatus according to claim 1, further comprising:

8. If the recording capacity of the recording medium is insufficient to record the third image data generated by the generation means using the recording means, the deletion means deletes the third image data recorded on the recording medium in order of lowest priority as set by the setting means. The image processing apparatus according to claim 7, characterized in that the recording means records the third image data generated by the generation means onto the recording medium after the recording capacity for recording has been secured by deletion by the deletion means.

9. The image processing apparatus according to claim 1, characterized in that the generation means generates the third image data while adjusting the range to be cut out from the first image data according to the available capacity of the recording medium.

10. The image processing apparatus according to claim 9, wherein the generation means generates the third image data so as to include the region in which the image quality defect is detected and the region surrounding the region, and adjusts the size of the surrounding region according to the available capacity of the recording medium.

11. The image processing apparatus according to claim 1, characterized in that the detection means changes the criteria for detecting poor image quality according to the available capacity of the recording medium.

12. The image processing apparatus according to claim 1, further comprising a deletion means for deleting the first image data from the recording medium when the recording means records the third image data on the recording medium if the first image data is recorded on the recording medium.

13. The image processing apparatus according to claim 1, characterized in that the detection means detects at least one of noise, black crushing, white clipping, and artifacts as the image quality defect.

14. An acquisition step of acquiring first image data and second image data obtained by applying image processing to the first image data, A detection step for detecting areas of poor image quality from the second image data, A generation step of generating a third image data that includes the region where the image quality defect was detected by cutting out the first image data, A recording step of recording the third image data in association with the second image data onto a recording medium, A control method for an image processing apparatus, characterized by comprising:

15. An acquisition step of acquiring first image data and second image data obtained by applying image processing to the first image data, A detection step for detecting areas of poor image quality from the second image data, A generation step of generating a third image data that includes the region where the image quality defect was detected by cutting out the first image data, A recording step of recording the third image data in association with the second image data onto a recording medium, A program that causes a computer to execute something.

Images