Image processing method, image processing device, and program

Abstract

To perform highly accurately at high speed blur determination of captured images.SOLUTION: An image processing method includes: a first step S11 of acquiring compressed image information including compressed image data generated by compressing a captured image through compression processing including encoding processing and encoding information used in the encoding processing; a second step S12 of generating decoded data by decoding the compressed image data using the encoding information; and a third step S13 of evaluating a blur of the captured image based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an image processing method for determining blur in an image. [Background technology]

[0002] There is a known method for determining whether or not a captured image is blurred due to camera shake or focus loss during capture (blur determination) based on DCT coefficients obtained by subjecting a captured image to frequency transformation such as Discrete Cosine Transform (DCT). A captured image obtained using an imaging device is generally compressed in JPEG (Joint Photographic Experts Group) format. In the JPEG format, an image is divided into a plurality of blocks of a predetermined size, and the captured image is compressed by frequency transformation, quantization processing, and encoding processing to generate compressed image data. When performing blur determination on a captured image using compressed image data, it is necessary to reconstruct the original captured image from the compressed image data, and then convert the image from the spatial domain to the frequency domain by DCT or the like, which takes time to perform blur determination.

[0003] Patent Document 1 discloses an image processing method in which blur determination is performed before a captured image is reconstructed in order to reduce the processing time required for blur determination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-220009 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of Patent Document 1, a decoding process and an inverse quantization process are performed on compressed image data compressed in JPEG format. Then, an evaluation value used for blur determination is calculated based on low-frequency information in the DCT coefficients after the inverse quantization process. However, when blur occurs due to camera shake or focus loss during image capture, the change in the captured image is mainly on the high-frequency side, so there is a problem that accuracy is low when only the low-frequency information is used.

[0006] Therefore, an object of the present invention is to perform blur determination of a captured image with high accuracy and high speed. [Means for solving the problem]

[0007] The image processing method of the present invention includes a first step of acquiring compressed image information including compressed image data generated by compressing a captured image through a compression process including an encoding process and encoding information used in the encoding process, a second step of generating decoded data by decoding the compressed image data using the encoding information, and a third step of evaluating blur of the captured image based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data. Effect of the Invention

[0008] According to the present invention, blur determination of a captured image can be performed with high accuracy and high speed. [Brief description of the drawings]

[0009] [Figure 1] Block diagram [Diagram 2] Zigzag scan diagram [Diagram 3] A diagram showing luminance information corresponding to blocks. [Figure 4] Diagram showing blocks after DCT [Diagram 5] A diagram showing the quantization table [Figure 6] FIG. 1 is a block diagram showing the first embodiment. [Figure 7]Block diagram of an imaging device according to a first embodiment [Figure 8] FIG. 1 is an external view of an imaging device according to a first embodiment. [Figure 9] FIG. 1 is a diagram showing a flow of blur determination processing in the first embodiment. [Figure 10] Histogram used for blur determination [Figure 11] Histogram used for blur determination [Figure 12] FIG. 13 is a diagram showing a flow of blur determination processing in the second embodiment. [Figure 13] A table of weights for improving symmetry. [Figure 14] Diagram showing quantization tables with improved symmetry [Figure 15] FIG. 1 shows a block diagram according to a second embodiment. [Figure 16] A table of weights for improving symmetry. [Figure 17] Diagram showing quantization tables with improved symmetry [Figure 18] Diagram showing areas in a block DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are given to the same members, and duplicated explanations will be omitted.

[0011] First, the gist of an embodiment of the present invention will be described. The image processing method in this embodiment evaluates the blur of a captured image based on at least one of the number of zero elements and the number of non-zero elements contained in the decoded data. The decoded data is generated only by decoding the compressed image data without requiring processes such as inverse quantization required for reconstructing the captured image. Therefore, the processing time for blur determination can be shortened compared to a method in which blur determination is performed after reconstructing the original captured image from the compressed image data.

[0012] In addition, when blur occurs in a captured image, the high-frequency components of the captured image change significantly, so the accuracy of the blur determination can be improved by performing the blur determination based on the high-frequency components of the captured image. In this embodiment, the number of zero elements and the number of non-zero elements included in the decoded data are correlated with the amount of high-frequency components included in the captured image. Therefore, by evaluating the blur based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data, the blur determination of the captured image can be performed with high accuracy.

[0013] Here, an overview of JPEG files will be described. JPEG files are used when capturing images compressed in JPEG format and storing them in a storage medium, and contain information about the captured images compressed in JPEG format (compressed image data) and information about the compression process.

[0014] JPEG files are divided into multiple segments by two-byte codes called markers. A segment is an area that groups together information recorded within a JPEG file by type. A JPEG file begins with an SOI (Start of Image) marker and ends with an EOI (End of Image) marker, with multiple segments and compressed image data in between. A segment has a marker that indicates the type of segment, information about the length of the segment, and recorded information. By comparing the marker information stored at the beginning of each segment, it is possible to obtain only the necessary segments.

[0015] Next, the Define Quantization Table (DQT) segment, the Define Huffman Table (DHT) segment, the Start of Frame (SOF) segment, and the Start of Scan (SOS) segment will be described.

[0016] The DQT segment holds a quantization table (quantization information) used in the quantization process in the compression process. A plurality of quantization tables can be held in the DQT segment, and different quantization tables can be used, for example, depending on the color. The quantization table has the same size as the blocks in the captured image. In this embodiment, the blocks are obtained by dividing the captured image into blocks of a predetermined size during the compression process in the JPEG format. In the case of the compression process in the JPEG format, the display format of the image colors is the YCbCr format converted from RGB (YCbCr conversion), so that the quantization tables used for Y (luminance information), Cr (blue color difference information), and Cb (red color difference information) can be held separately.

[0017] The DHT segment holds the Huffman table (encoding information) used in Huffman encoding, which is an encoding process in the JPEG format. The DHT segment can define multiple tables, just like the DQT segment. Generally, the DHT segment holds four tables in total: DC (direct current) and AC (alternating current) components for luminance information, and DC and AC components for blue and red color difference information. If a block has N x N pixels, the DC component is 1 / N of the total within the block. In other words, when the DC component becomes zero, it means that the corresponding block part of the captured image is all zero. Therefore, the DC component can be zero only when the AC component is also zero, so it is possible to perform blur determination without obtaining information on the DC component. Note that Huffman encoding differs between the DC component and the AC component. The DC component is converted into a bit string for the quantized DCT coefficient, and then a Huffman code corresponding to the number of bits required to express the value is added to the front of the bit string to create a single code. On the other hand, AC components are processed in the same way as DC components, in that a Huffman code is assigned for each bit required to represent a number, but run-length compression is then performed. Therefore, the Huffman table for AC components stores not only the bit data length, but also a run-length value, which indicates how many consecutive zeros there are. As a result, if a number is followed by multiple zeros, they are all represented by a single Huffman code.

[0018] The SOF segment holds information such as the size and sampling ratio of the captured image.

[0019] The SOS segment indicates that compressed image data follows and is stored before the compressed image data. The compressed image data is stored in units called MCU (Minimum Coded Unit). When there are three colors and the sampling ratio is 4:2:0, the MCU consists of four 8x8 pixel blocks for Y, one 8x8 pixel block for Cr, and one 8x8 pixel block for Cb, for a total of six blocks. In this case, since Cr and Cb also originally have four 8x8 pixel blocks like Y, 12 blocks of data are thinned out to reduce the number of blocks by six. In this way, when the sampling ratio is 4:2:0, the Cr and Cb data is thinned out to one-fourth of that of Y. Incidentally, when the sampling ratio is 4:4:4, there is no thinning out of Cr and Cb, and the number of data is the same as that of Y.

[0020] Next, the blocks will be described with reference to FIG. 1. FIG. 1 is an image diagram of the blocks in this embodiment, with the horizontal frequency axis μ and the vertical frequency axis ν. The captured image in this embodiment is YCbCr converted, with a sampling ratio of 4:2:0. In this case, as described above, the MCU is composed of 12 blocks, of which 4 are blocks of the luminance component Y. Each block is composed of 8×8 pixels. The frequency (μ,ν)=(0,0) is the DC component, and the remaining 63 data are the AC components. The higher the values ​​of the frequencies μ and ν, the higher the frequency of the component.

[0021] Although FIG. 1 shows two-dimensional data, the data actually stored is one-dimensional. A process called zigzag scan is used to convert two-dimensional data into one-dimensional data. FIG. 2 is an explanatory diagram of zigzag scan, and the numbers written in each square indicate the order of storage when converted into one-dimensional data. As shown in FIG. 2, the AC components start from (μ,ν)=(1,0) and the two-dimensional data proceeds from the low frequency side to the high frequency side, ending with the 63rd (μ,ν)=(7,7). Since the one-dimensional data after zigzag scan is arranged in order of frequency, zeros tend to line up, especially in the latter half of the high frequency side data. In the JPEG format, high compression is achieved by performing an encoding process on such data with continuous zeros. In order to obtain two-dimensional block data as shown in FIG. 1, it is necessary to convert the one-dimensional data into two-dimensional data based on the information on the storage order in FIG. 2. However, in the blur determination of the captured image in this embodiment, it is sufficient to obtain information on the elements in the block whose DCT coefficients are zero, so two-dimensionalization is not essential.

[0022] Here, the quantization process will be described with reference to Figs. 3 to 6. Fig. 3 shows an image diagram of luminance information of 8 x 8 pixels in a captured image. Each value in Fig. 3 represents the luminance value of the captured image, and is generated by dividing the captured image into blocks of a predetermined size. When compressing in JPEG format, frequency conversion is performed for each block of 8 x 8 pixels. Note that the frequency conversion in this embodiment uses discrete cosine transform (DCT). Also, in this embodiment, only luminance information generated by color conversion is used, but this is not limited to this.

[0023] Figure 4 shows the DCT coefficients when a discrete cosine transform is performed on the luminance information in Figure 3. The DCT coefficients in Figure 4 have coordinates corresponding to those in Figure 1, and represent the frequency distribution in an 8 x 8 pixel block contained in a captured image. The DCT coefficients show the DC component in the upper left and AC components elsewhere, with the value decreasing as the frequency increases.

[0024] FIG. 5 shows a quantization table in this embodiment. The quantization table has 8×8 pixels, just like the block. Each element of the quantization table has a value between 1 and 255. The DCT coefficients can be quantized by dividing each value of the DCT coefficient and each value of the quantization table at the corresponding position. Therefore, the closer each element of the quantization table is to 1, the smaller the effect of quantization becomes. When a quality level can be set in creating a JPEG image, the quality level can be maximized by setting all elements of the quantization table to 1.

[0025] However, even if all the elements of the quantization table are 1, they are converted to integers after quantization, so that, for example, in the state of Figure 4, all values ​​with absolute values ​​less than 0.5 become zero. In this way, the JPEG format is essentially a lossy conversion because information is lost due to the quantization process. Note that when the quality level is set to the minimum, all the elements in Figure 5 become 255, and most of the DCT coefficients after quantization become zero, which increases the compression rate.

[0026] FIG. 6 shows the DCT coefficients after quantization. The quantized DCT coefficients in FIG. 6 are calculated by dividing the DCT coefficients in FIG. 4 by the values ​​in the quantization table in FIG. 5 and rounding off. The quantized DCT coefficients tend to have non-zero values ​​on the low frequency side in the upper left, and zero values ​​on the high frequency side in the lower right. When the two-dimensional data in FIG. 6 is converted into one dimension by zigzag scanning, the elements that are non-zero are up to the 24th (μ,ν)=(3,3) AC component, and the elements that are zero are from the 25th onwards. In this case, the number of zero elements (evaluation value) in FIG. 6 is 43. For example, by comparing the evaluation value with a threshold, it is possible to determine whether or not there is blur in the captured image. Details of determining blur in a captured image will be described later.

[0027] Furthermore, by performing the above-mentioned encoding process on the quantized DCT coefficients, compressed image data that is saved as a JPEG file can be generated.

[0028] In this embodiment, the evaluation value is calculated based on decoded data equivalent to the DCT coefficients after quantization when compressing the captured image. The decoded data is generated by decoding the compressed image data using a quantization table. By performing the evaluation using the decoded data without performing the inverse quantization process, the processing load in the blur determination can be reduced.

[0029] The image processing method described above is merely an example, and the present invention is not limited to this. Details of other image processing methods will be described in the following examples.

[0030] [Example 1] Next, the configuration of the imaging device 100 in the first embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a block diagram showing the configuration of the imaging device 100. Fig. 8 is an external view of the imaging device 100. An image processing program for performing blur determination in this embodiment is installed in the imaging device 100.

[0031] The imaging device 100 in this embodiment is intended to be used for image inspection, particularly for inspection of the wall surface of a structure such as a bridge, and for visual inspection of parts. When performing such image inspection, an inspection method that automatically recognizes cracks, scratches, etc. is often used. In general, cracks and scratches are often detected by using pattern recognition, but these methods are based on the premise that the image is not blurred. Here, blurring includes both out-of-focus and camera shake. Out-of-focus refers to a state in which the subject (the wall surface of the structure in the case of a bridge) is out of focus and is blurred. Camera shake refers to the blurring of a captured image caused by the movement of the imaging device body during exposure when shooting. For example, when a user shoots while holding the imaging device in his / her hand, if the imaging device moves vertically during exposure, the captured image will be significantly blurred vertically. In this embodiment, camera shake is not limited to when a person holds the imaging device. For example, in an imaging device mounted on an unmanned aircraft such as a drone, camera shake also includes blurring caused by shaking during flight. If the captured image used in such image inspection contains blur, the accuracy of crack identification will decrease.

[0032] Therefore, a process is performed to determine whether or not the captured image is blurred using the blur judgment described below, and if there is an image that is judged to be "blurred," for example, that image is excluded and an image inspection is performed, or the image is retaken, thereby improving the accuracy of identification.

[0033] The imaging device 100 shown in FIG. 7 includes an imaging section (imaging means) 101, an image processing section (processing means) 102, a storage section 103, a display section 104, and a system controller 105.

[0034] The imaging unit 101 has an optical system 101a that collects light incident from the subject space and an imaging element 101b having a plurality of pixels, and obtains an image of the subject space as a captured image. The imaging element 101b is composed of, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The obtained captured image is input to the image processing unit 102. The image processing unit 102 has a decoding unit 102a, an evaluation unit 102b, and a blur determination unit 102c. The decoding unit 102a decodes the encoded DCT coefficients. The evaluation unit 102b calculates an evaluation value for blur determination using the decoded DCT coefficients (decoded data). The blur determination unit 102c determines whether the captured image is blurred based on the evaluation value. Details of this process will be described later. The functions of the image processing unit 102 can be realized by one or more processors (processing means). The captured images acquired by the imaging unit 101 are stored in the storage unit 103 in JPEG format. The format in which the captured images are stored is not limited to this. In this embodiment, the blur of the captured images stored in the storage unit 103 is evaluated. The display unit 104 can display the blur determination result and the captured image. The above series of controls are performed by the system controller 105.

[0035] Next, the blur determination process performed in the image processing unit 102 will be described with reference to Fig. 9. Fig. 9 is a flowchart relating to the blur determination process.

[0036] In step S11 (first step), the image processing unit 102 acquires compressed image information from the storage unit 103. In this embodiment, the compression is performed in JPEG format. In the JPEG format, it is possible to acquire only the necessary segments by checking marker information indicating the type of the segment. In this embodiment, information on the SOS segment, the DHT segment, and the SOF segment is acquired as the compressed image information. The SOS segment holds image data compressed in JPEG format (compressed image data). The DHT segment holds a Huffman table (encoding information) used for Huffman encoding, which is used for decoding data in the following steps. In addition, the SOF segment holds information such as the size of the image and the sampling ratio. In this embodiment, the evaluation value is calculated based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data. At this time, it is not necessary to acquire a table of DC components from the DHT segment.

[0037] In this embodiment, the compressed image information is acquired from the imaging device 100, but is not limited thereto. For example, compressed image information based on an image acquired using another imaging device may be acquired. As described above, the captured image is preferably an image of a wall surface of a structure such as a bridge or an exterior of a part. With such an image, the subject included in the image is limited to a certain extent, so that the accuracy of blur determination can be improved.

[0038] In step S12 (second step), the decoding unit 102a decodes the acquired compressed image data. The compressed image data has been subjected to encoding processing using encoding information during compression processing, and the decoding unit 102a decodes the compressed image data using the encoding information. Alternatively, only compressed image data based on luminance information obtained by YCbCr conversion from the captured image may be decoded. By determining blur using decoded data obtained by decoding only the luminance information, the processing speed can be increased.

[0039] In step S13 (third step), the evaluation unit 102b calculates an evaluation value used for blur determination. In this embodiment, the evaluation value is calculated by counting the number of zero elements in each block of the decoded data. Since the number of elements included in each block of the quantized DCT coefficients is a specific number, the same result can be obtained even if the number of elements other than zero is used as the evaluation value. Therefore, the evaluation value is not limited to this, and the number of elements other than zero may be used as the evaluation value.

[0040] The quantized DCT coefficient in this embodiment is a block of 8×8 pixels, and has 64 elements. The number of zero elements correlates with the blur of the captured image, and when the number of zero elements is small, the block contains many high-frequency components. On the other hand, when the number of zero elements is large, the block contains few high-frequency components. When the number of zero elements included in the decoded data is large, it means that either an object with little fine structure (few high-frequency components) is captured, or high-frequency components are lost due to camera shake or out of focus. Thus, since the number of zero elements varies depending on the object, in this embodiment, blur determination is performed based on information of multiple blocks in order to improve the accuracy of blur determination.

[0041] In step S14 (fourth step), the blur determination unit 102c performs blur determination based on the evaluation values. In this embodiment, the blur determination unit 102c performs blur determination using a frequency distribution (histogram) of the acquired evaluation values.

[0042] Histograms of evaluation values ​​for each block for different images are shown in Figs. 10 and 11. The vertical axis in Figs. 10 and 11 is the count number, and the horizontal axis is the evaluation value. Note that since calculations are performed for blocks of 8x8 pixels, the maximum value of the horizontal axis including DC components is 64. Also, although Figs. 10 and 11 are calculated using only the DCT coefficient of luminance information Y, this is not limiting. For example, color difference information Cr, Cb may also be used. Note that the horizontal axis in Figs. 10 and 11 is just an example, and the number of elements may be counted for each finer (or coarser) range.

[0043] The number of pixels in the images used in the examples is 6720 x 4480 pixels. If an evaluation value is calculated for each 8 x 8 pixel block for the entire image, an image (map) having 470,400 evaluation values, which is 1 / 64, can be generated. Therefore, Figures 10 and 11 are histograms created from 470,400 pieces of data.

[0044] In this embodiment, blur determination is performed using the mode obtained using a histogram. The mode in FIG. 10 is 38, and the mode in FIG. 11 is 46. That is, FIG. 11 has more zeros in the DCT coefficients than FIG. 10, so it is highly likely that it is blurred. In the blur determination in this embodiment, a threshold value is set and the magnitude is compared with the value. For example, when the threshold value is set to 40, FIG. 10 is determined to be "not blurred" and FIG. 11 is determined to be "blurred". In this embodiment, the threshold value is not limited to this, and may be set according to the object of imaging. For example, when imaging an object with many fine structures (containing many high-frequency components), it is preferable to set the threshold value high. On the other hand, when imaging an object with few fine structures (containing few high-frequency components), it is preferable to set the threshold value low. In this way, the threshold value is preferably set according to the object of imaging, and the threshold value may be calculated in advance based on the image captured.

[0045] In addition, in this embodiment, the most frequent value is used as an index for blur determination, but an index such as a median, an average value, or a total number may be used. In addition, in this embodiment, the evaluation value is calculated from decoded data corresponding to the entire captured image, but an evaluation value calculated from decoded data (partial decoded data) corresponding to a partial area of ​​the captured image may be calculated. For example, when multiple images of a bridge wall are taken for inspection purposes, an image including pipes and background that are not the object of inspection may be obtained. In this case, the accuracy of blur determination can be improved by calculating the evaluation value excluding the area other than the object of inspection. In this embodiment, a two-dimensional map corresponding to the captured image is created and the map is used for evaluation, so that blur determination can be performed by cutting out only a partial area from the captured image by, for example, a trimming process.

[0046] Furthermore, blur determination may be performed using thumbnail image data instead of normal compressed image data. In addition to compressed image data, compressed image information may also store thumbnail image (reduced image) data with fewer pixels than the main compressed image data. By performing blur determination using thumbnail image data with fewer pixels, the blur determination process can be performed more quickly.

[0047] Although an example of calculating an evaluation value based on the number of zero elements or the number of non-zero elements included in the decoded data has been shown, the present invention is not limited to this. In the present embodiment, the blur judgment may be performed by evaluating the blur of the captured image based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data. For example, the blur judgment may be performed using the ratio between the number of zero elements and the number of non-zero elements or the difference between the number of zero elements and the number of non-zero elements as the evaluation value.

[0048] This embodiment is characterized in that the blur of the captured image is evaluated based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data. The decoded data is generated from the compressed image data only by the decoding process without the need for processes such as inverse quantization required for reconstructing the captured image, so that the processing time can be reduced. In addition, the blur of the captured image is evaluated based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data, which are correlated with the amount of high-frequency components included in the captured image, so that the blur can be determined with high accuracy.

[0049] [Example 2] Next, a configuration of the image pickup device 100 in the second embodiment will be described. In this embodiment, the image pickup device 100 performs a process of determining the presence or absence of blur in the captured image, as well as determining the direction of the blur. By performing blur determination using the method of this embodiment, it is possible to determine directional blur, such as blur caused by camera shake.

[0050] Fig. 12 is a flow chart of the blur determination process in this embodiment. Steps S21 and S22 in Fig. 12 are similar to steps S11 and S12 in Fig. 9, so a description thereof will be omitted.

[0051] In step S23, the decoding unit 102a performs a process of symmetrizing the decoded DCT coefficients. Here, Fig. 13 shows a distribution obtained by transposing the quantization table in Fig. 5.

[0052] Generally, a quantization table is asymmetric in the horizontal and vertical directions. If the weights of the quantization table are asymmetric in the horizontal and vertical directions, it is not possible to perform a correct evaluation by using the quantized DCT coefficients as they are to compare the horizontal and vertical coefficients. For example, in Figure 4, the DCT coefficient of (μ,ν)=(5,0) is 2.09, and since the corresponding weight in Figure 5 is 2, it becomes 1 when rounded off after quantization. However, since the weight is 1 in Figure 5 for (μ,ν)=(0,5), it becomes 2 if quantization is performed using this weight. By dividing back this quantized DCT coefficient by the quantization table in Figure 5, it is possible to improve the symmetry (correct the asymmetry). However, if the quantized DCT coefficient is zero, it is not possible to perform division, and therefore the symmetry cannot be improved. As mentioned above, JPEG format data compresses the amount of information by actively increasing the number of elements that become zero, so ignoring the fact that the DCT coefficient becomes zero will result in a decrease in accuracy.

[0053] Therefore, in this embodiment, in order to improve the symmetry of the quantization table, the weights in FIG. 13 are further multiplied. FIG. 13 is a table of weights obtained by transposing the length and width of the table in FIG. 5. In addition, by multiplying the values ​​at corresponding positions in the quantization table in FIG. 5 and the table in FIG. 13, a quantization table with improved symmetry shown in FIG. 14 is generated. The quantization table in FIG. 14 has symmetry in the horizontal and vertical directions. By symmetrizing the quantization table in this way, it is possible to perform equal evaluation in the horizontal and vertical directions.

[0054] Next, a block (corrected decoded data) in which the symmetry of the quantization table has been improved is shown in Fig. 15. Fig. 15 shows the result of multiplying the decoded DCT coefficients in Fig. 6 by the table in Fig. 13 and rounding off the result.

[0055] The method of improving the symmetry of the quantization table is not limited to this. For example, by multiplying the weights shown in FIG. 16, the weights applied to FIG. 4 may be substantially the same as those shown in FIG. 17 in combination with the quantization table of FIG. 5. By adopting such a configuration, the value of the weight applied to FIG. 4 can be reduced. FIG. 16 is an example of a table for improving symmetry, and FIG. 17 is an example of a quantization table with improved symmetry. FIG. 17 is generated by taking the larger value at the corresponding position of the quantization table of FIG. 5 and the table of FIG. 13. In this case, if symmetrization is performed using a weight having a value less than 1, the value becomes zero, so compression is possible, but since data deteriorates with compression, it is desirable to perform symmetrization using a table configured with values ​​of 1 or more as shown in FIG. 16. In addition, in this embodiment, an example of multiplying weights so as to be symmetric in the horizontal and vertical directions has been shown, but symmetrization may be performed so as to have a rotationally symmetric distribution centered on the DC component.

[0056] In step S24, the evaluation unit 102b calculates an evaluation value used for blur determination. In this embodiment, the decoded data is two-dimensionally processed and expressed as a block of 8×8 pixels. Here, FIG. 18 shows a diagram of the area for determining the direction of blur. As shown in FIG. 18, the decoded data expressed in two dimensions of 8×8 pixels is divided into two areas of 4×4 pixels. Area A (first area) is an area where the horizontal direction is high frequency and the vertical direction is low frequency, and area B (second area) is an area where the horizontal direction is low frequency and the vertical direction is high frequency. In this embodiment, the first and second evaluation values ​​are calculated by counting the number of elements whose DCT coefficients are zero for each of area A and area B. As an example, when the DCT coefficients in FIG. 15 are used, the number of zero elements in area A is 13, and the number of zero elements in area B is 14. The number of elements other than zero may be used as the evaluation value, and in this case, the process can be performed in the same manner.

[0057] In step S25, the blur determination unit 102c performs blur determination based on the evaluation value. In this embodiment, the blur determination unit 102c performs blur determination using a frequency distribution (histogram) generated based on the evaluation value for each of the regions A and B in the decoded data.

[0058] The blur determination unit 102c in this embodiment can determine the direction of blur by comparing region A with region B. For example, if the mode in region A is greater than the mode in region B, it is determined that there is "blur" in the horizontal direction. On the other hand, if the mode in region A is smaller than the mode in region B, it is determined that there is "blur" in the vertical direction. Furthermore, the blur determination unit 102c may use a threshold value for the difference between region A and region B and determine that there is "no blur" if the difference in the mode between region A and region B is smaller than the threshold value.

[0059] In this embodiment, the method of evaluating the blur direction by dividing into two regions has been described, but the method may be performed by dividing into three or more regions. The blur direction may be evaluated based on the maximum coordinates of non-zero elements in the horizontal and vertical directions. For example, the maximum coordinates of non-zero elements in FIG. 15 are (μ,ν)=(5,0) in the horizontal direction and (μ,ν)=(0,5) in the vertical direction, and the maximum coordinates of non-zero elements in FIG. 6 are (μ,ν)=(5,0) in the horizontal direction and (μ,ν)=(0,6) in the vertical direction.

[0060] In this embodiment, the method of calculating evaluation values ​​corresponding to the first and second regions in the decoded data and determining the direction of blurring has been described. In particular, by adding a process for improving the symmetry of the quantization table in step S23, the accuracy of determining the direction of blurring can be improved.

[0061] (Other Examples) The present invention can also be realized by supplying a program for realizing one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions. The image processing device in the present invention may be any device having the image processing function of the present invention, and may be realized in the form of an imaging device or a PC.

[0062] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program that can perform blur determination of a captured image with high accuracy and high speed.

[0063] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0064] Embodiments of the present invention include the following methods and compositions.

[0065] [Method 1] a first step of acquiring compressed image information including compressed image data generated by compressing a captured image through a compression process including an encoding process and encoding information used in the encoding process; a second step of decoding the compressed image data using the encoding information to generate decoded data; and a third step of evaluating blur of the captured image based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data.

[0066] [Method 2] the compression process includes a process of generating luminance information of the captured image, The image processing method according to method 1, wherein in the second step, the decoded data is generated by decoding only data corresponding to the luminance information from the compressed image data.

[0067] [Method 3] The image processing method according to method 1 or 2, characterized in that in the third step, an evaluation value is calculated based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data, and the evaluation value is compared with a threshold value to determine whether or not the captured image is blurred.

[0068] [Method 4] the compression process includes a frequency transform applied to each of a plurality of blocks of a predetermined size of the captured image; 4. The image processing method according to any one of Methods 1 to 3, wherein in the second step, a plurality of decoded data corresponding to each of the plurality of blocks is generated.

[0069] [Method 5] The image processing method according to Method 4, characterized in that in the third step, a plurality of evaluation values ​​are calculated based on at least one of the number of zero elements and the number of non-zero elements contained in each of the plurality of decoded data, and the presence or absence of blurring of the captured image is determined based on at least one of the mode, average, median, and sum of the plurality of evaluation values.

[0070] [Method 6] The image processing method according to any one of Methods 1 to 5, characterized in that in a third step, the blur of the captured image is evaluated based on at least one of the number of zero elements and the number of non-zero elements contained in partial decoded data of the decoded data corresponding to a partial area of ​​the captured image.

[0071] [Method 7] 7. The image processing method according to any one of Methods 1 to 6, wherein in the second step, decoded data is generated without performing inverse quantization.

[0072] [Method 8] The compression process includes a quantization process, the compressed image information includes quantization information related to the quantization process, The image processing method according to any one of methods 1 to 7, further comprising a step of generating corrected decoded data by correcting asymmetry in the quantization process based on the decoded data and the quantization information.

[0073] [Configuration 1] 9. An image processing device comprising a processing means capable of executing the image processing method according to any one of Methods 1 to 8.

[0074] [Configuration 2] A processing means capable of executing the image processing method according to any one of methods 1 to 8; and imaging means for acquiring an image of a subject space as a photographed image.

[0075] [Configuration 3] A program for causing a computer to execute the image processing method according to any one of Methods 1 to 8.

[0076] [Configuration 4] A storage medium storing the program according to configuration 3. [Explanation of symbols]

[0077] Step S11: First process Step S12: Second process Step S13: Third process

Claims

1. a first step of acquiring compressed image information including compressed image data generated by compressing an image through a compression process including an encoding process and encoding information used in the encoding process; a second step of decoding the compressed image data using the encoding information to generate decoded data; and a third step of evaluating blur of the image based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data.

2. the compression process includes a process of generating luminance information of the image; 2. The image processing method according to claim 1, wherein in the second step, the decoded data is generated by decoding only data corresponding to the luminance information from the compressed image data.

3. 2. The image processing method according to claim 1, wherein in the third step, an evaluation value is calculated based on at least one of the number of zero elements and the number of non-zero elements included in the decoded data, and the presence or absence of blurring of the image is determined by comparing the evaluation value with a threshold value.

4. the compression process includes a frequency transform applied to each of a plurality of blocks of a predetermined size of the image; 2. The image processing method according to claim 1, wherein in the second step, a plurality of decoded data corresponding to each of the plurality of blocks is generated.

5. 5. The image processing method according to claim 4, wherein in the third step, a plurality of evaluation values ​​are calculated based on at least one of the number of zero elements and the number of non-zero elements included in each of the plurality of decoded data, and the presence or absence of blurring of the image is determined based on at least one of a mode, an average, a median, and a sum of the plurality of evaluation values.

6. 6. The image processing method according to claim 1, wherein in the third step, the blur of the image is evaluated based on at least one of the number of zero elements and the number of non-zero elements contained in partial decoded data corresponding to a part of the image in the decoded data.

7. 6. The image processing method according to claim 1, wherein in the second step, decoded data is generated without performing inverse quantization.

8. the compression process includes a quantization process; the compressed image information includes quantization information related to the quantization process; The image processing method according to claim 1 , further comprising the step of generating corrected decoded data by correcting asymmetry in the quantization process based on the decoded data and the quantization information.

9. 6. An image processing apparatus comprising processing means capable of executing the image processing method according to claim 1.

10. 6. An imaging device comprising: a processing unit capable of executing the image processing method according to claim 1; and an imaging unit for acquiring an image of a subject space as a photographed image.

11. A program causing a computer to execute the image processing method according to any one of claims 1 to 5.

12. A storage medium storing the program according to claim 11.

Images