Image processing apparatus and method

Abstract

The invention discloses an image processing device and method. The method comprises the following steps: marking at least one first periodic image block with a periodic characteristic in a zoomed-out current image by at least one first mark; performing a first motion estimation on the reduced current image and a reduced reference image based on the at least one first mark to generate a plurality of first motion vectors; marking at least one nth period image block with another periodic characteristic in the current image by at least one nth mark; performing an nth motion estimation on the current image and a reference image based on the at least one nth mark to generate a plurality of nth motion vectors; a motion compensation is performed based on the nth motion vectors to generate a frame-complemented image.

Description

Technical Field

[0001] The present application relates to an image processing apparatus and method, and particularly to an image processing apparatus and method for motion estimation (ME) and motion compensation (MC). Background Art

[0002] In the field of image processing, when performing frame rate conversion (FRC), motion vectors can be calculated through motion estimation, and after processing, they are then handed over to motion compensation to generate interpolated images between two original images, so that the images are smoother. However, when performing motion estimation, if there are repetitive images in the image, such as: blinds, striped shirts, windows of office buildings, containers at the dock, etc., it will cause misjudgment of motion estimation and result in periodic broken or repeat broken phenomena, and then calculate incorrect motion vectors. When this phenomenon occurs, unnatural broken images will appear in the image, resulting in a reduced viewing experience. Further, when detecting whether there are repetitive images in the image, if it is necessary to detect periodic images of various sizes and ranges, the computational burden will increase.

[0003] In view of this, how to detect the positions where repetitive images appear in the image and correct the results of motion estimation during the execution of motion estimation, while taking into account periodic images of various sizes and ranges as well as computational efficiency, is an urgent goal that the industry needs to strive for.

[0004] Summary of the Invention

[0005] ​​To solve the above problems, the present application proposes an image processing apparatus, including a memory and a processor. The memory is used to store a current image and a reference image. The processor is coupled to the memory and is configured to perform the following operations: reducing the current image and the reference image to respectively generate a reduced current image and a reduced reference image; marking at least one of a plurality of first blocks in the reduced current image with at least one first mark as at least one first periodic image block, where the at least one first periodic image block has a first periodic feature; performing a first motion estimation on the reduced current image and the reduced reference image based on the at least one first mark to generate a plurality of first motion vectors; marking at least one of a plurality of nth blocks in the current image with at least one nth mark as at least one nth periodic image block, where the at least one nth periodic image block has an nth periodic feature; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the at least one nth mark to generate a plurality of nth motion vectors; and performing a motion compensation on the current image and the reference image based on the nth motion vectors to generate an interpolated image between the current image and the reference image.

[0006] The present application also proposes an image processing method applicable to an electronic device. The steps include: reducing a current image and a reference image to respectively generate a reduced current image and a reduced reference image; marking at least one of a plurality of first blocks in the reduced current image with at least one first mark as at least one first periodic image block, where the at least one first periodic image block has a first periodic feature; performing a first motion estimation on the reduced current image and the reduced reference image based on the at least one first mark to generate a plurality of first motion vectors; marking at least one of a plurality of nth blocks in the current image with at least one nth mark as at least one nth periodic image block, where the at least one nth periodic image block has an nth periodic feature; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the at least one nth mark to generate a plurality of nth motion vectors; and performing a motion compensation on the current image and the reference image based on the nth motion vectors to generate an interpolated image between the current image and the reference image.

[0007] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are intended to provide further explanation of the present application as claimed. Description of the Drawings

[0008] To make the above and other objects, features, advantages and embodiments of the present application more apparent and understandable, the description of the accompanying drawings is as follows: Figure 1 Schematic diagram for generating a compensated image between two images for motion estimation and motion compensation operations; Figure 2 Schematic diagram of an image processing apparatus in some embodiments of the present application; Figure 3 Flowchart of an image processing method in some embodiments of the present application; Figure 4 Schematic diagram of a motion estimation operation in some embodiments of the present application; Figure 5 Schematic diagram of reducing an image in some embodiments of the present application; Figure 6 Flowchart of an operation for marking periodic image blocks in some embodiments of the present application; Figure 7 Schematic diagram of translating and comparing pixels in an image in some embodiments of the present application; Figure 8 Flowchart of another operation for marking periodic image blocks in some embodiments of the present application; Figure 9 Flowchart of a scanning operation in motion estimation in some embodiments of the present application; Figure 10 Flowchart of another scanning operation in motion estimation in some embodiments of the present application; and Figure 11 Flowchart of an operation for selecting a motion vector from candidate vectors in some embodiments of the present application. Detailed Description of the Invention

[0009] To make the description of the present application more detailed and complete, reference may be made to the accompanying drawings and the following various embodiments, where the same numbers in the drawings represent the same or similar components.

[0010] Motion estimation and motion compensation are used to generate a compensated image between two images to increase the frame rate. For example, please refer to Figure 1 , image Fk-1 and image Fk are two adjacent frames in a video. Motion estimation and motion compensation are used to generate compensated frames FC1 to FC4 between image Fk-1 and image Fk. When performing motion estimation, image Fk-1 and image Fk can be divided into i by j blocks, and a three-dimensional recursive search (3D recursive search) is used to find the best motion vector for each block. vector).

[0011] More specifically, the three-dimensional recursive search involves multiple scan operations. Each scan operation generates candidates for various motion vectors within the search window for each block of the entire image, based on the initial vector of the block and the characteristics of various image changes. For example, candidates such as zero, spatial, temporal, random, global, etc. are generated, and the one with the highest matching degree is calculated as the motion vector of this block. Then, when the scan operation is performed again, the scan operation adds a random vector to the motion vector of each block obtained in the previous scan as the initial vector of this block. Thus, through multiple scans, the best motion vector for each block (such as the motion vectors MV1 to MVn described later) can be converged.

[0012] After that, motion compensation generates the compensated frame images FC1 to FC4 between image Fk-1 and image Fk based on image Fk-1, image Fk, and the best motion vector of each block. For example, if the best motion vector is roughly a vector from the lower left to the upper right, then based on the circle in the lower left corner of image Fk-1 and the circle in the upper right corner of image Fk, the multiple circles in the compensated frame images FC1 to FC4 will be arranged in sequence from the lower left to the upper right. In other words, through the operations of motion estimation and motion compensation, the compensated frame images FC1 to FC4 can be generated based on image Fk-1 and image Fk.

[0013] To avoid the compensated image being fragmented due to repetitive images in the image during motion estimation, it can be first determined whether there are blocks with periodic images in the image before performing motion estimation and motion compensation. For example, first, a processor calculates the multiple gray-scale differences between multiple adjacent pixels in image Fk-1 to confirm the change of pixel values and find multiple peak pixels and multiple valley pixels of the pixel values. In some embodiments, a peak pixel refers to a pixel at the turning point where the gray-scale value changes from gradually increasing to gradually decreasing among multiple continuously arranged pixels. In some embodiments, a valley pixel refers to a pixel at the turning point where the gray-scale value changes from gradually decreasing to gradually increasing among multiple continuously arranged pixels. Then, the processor calculates the multiple peak-to-peak distances between these peak pixels and the multiple valley-to-valley distances between these valley pixels, and counts whether there is periodicity between these peak-to-peak distances and these valley-to-valley distances (for example: these peak-to-peak distances are similar to each other and these valley-to-valley distances are similar to each other). Finally, the processor marks the blocks in image Fk-1 where there is periodicity between these peak-to-peak distances and these valley-to-valley distances.

[0014] To further detect whether there are periodic images in the image and perform motion estimation and motion compensation on the image, the present application proposes an image processing device. Please refer toFigure 2 , which is a schematic diagram of the image processing device 1 in the first embodiment of the present application. As Figure 2 shown, the image processing device 1 includes a processor 12 and a memory 14, where the processor 12 is coupled to the memory 14.

[0015] In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit, a multi-processor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable computing unit.

[0016] The memory 14 is used to store the current image Fn and the reference image Ffn. In some embodiments, the memory 14 may include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), hard disks, and / or optical discs.

[0017] Please further refer to Figure 3 , which is a flowchart of the image processing method 200 in an embodiment of the present application, where the image processing method 200 includes steps S21 to S26. The image processing device 1 is used to execute the image processing method 200 to perform motion estimation and motion compensation. Before the image processing device 1 generates a motion vector, it will first detect whether there is a periodic image in the image and the position where the periodic image appears.

[0018] In step S21, the processor 12 of the image processing device 1 reduces the current image Fn and the reference image Ffn to respectively generate a plurality of reduced current images F1 to Fn-1 with different resolutions and a plurality of reduced reference images Ff1 to Ffn-1 with different resolutions.

[0019] Specifically, please refer to FIGS. 4 and 5, Figure 4 which is a schematic diagram of the motion estimation operation MEP in an embodiment of the present application, Figure 5 which is a schematic diagram of reducing an image in an embodiment of the present application. The image Fn-1 and the image Ffn-1 are respectively the images obtained by reducing the size of the current image Fn and the reference image Ffn once. By analogy, the images F1 and Ff1 are respectively the images obtained after reducing the size n-1 times. For example, the magnification of reducing the size can be 1 / 2 times or 1 / 4 times, but the present case is not limited thereto. In one embodiment, the current image Fn is Figure 1The image Fk-1, and the reference image Ffn is Figure 1 The image Fk, that is, the reference image Ffn and the current image Fn can be two adjacent frames in a continuous image (e.g., a video), but the present case is not limited thereto.

[0020] In step S22, the processor 12 of the image processing device 1 marks at least one of the multiple first blocks of the scaled-down current image F1 with at least one first mark as at least one first periodic image block, where the at least one first periodic image block has a first periodic feature.

[0021] Specifically, before performing the first motion estimation 1ME, the processor 12 first marks the blocks (i.e., the first periodic image blocks) in the image F1 that have periodic images (e.g., images with repeating patterns such as blinds, striped shirts, windows of office buildings, containers at a dock, etc.), where the first mark indicates the blocks in the image F1 that have periodic images.

[0022] In some embodiments, step S22 further includes steps S221 to S224. The processor 12 can mark the first periodic image blocks through steps S221 to S224. For ease of explanation, the flowcharts of steps S221 to S224 are shown in Figure 6 . Figure 7 It is a grayscale schematic diagram of multiple graphics according to an embodiment of the present application document. The following will be described in conjunction with Figure 7 to illustrate Figure 6 Steps S221 to S224. In step S221, the processor 12 compares the graphic P in the image F1 with multiple translated graphics P1, P2, and P3 to calculate multiple grayscale differences. Specifically, the graphic P is formed by multiple pixels arranged continuously in the image F1. The processor 12 moves the graphic P in the same direction by different translation amounts to obtain multiple translated graphics P1, P2, and P3. In some embodiments, the translation amounts corresponding to the translated graphics P1, P2, and P3 are M pixels, 2M pixels, and 3M pixels respectively, where M is a positive integer. The processor 12 calculates the grayscale differences of the pixels in the overlapping parts between the graphic P and each of the graphics P1, P2, and P3.

[0023] For example, when M is 1, the calculation method of the grayscale difference between the graphic P and the translated graphic P1 is as follows: (1) Calculate the absolute value of the grayscale of the second pixel of the graphic P minus the grayscale of the first pixel of the translated graphic P1, and calculate the absolute value of the grayscale of the third pixel of the pixel P minus the grayscale of the second pixel of the pixel P1, and so on; and (2) Sum up and average the aforementioned multiple absolute values to obtain the grayscale difference between the graph P and the translated graph P1. The calculation methods of the grayscale differences between the graph P and the translated graphs P2 and P3 are similar to the above, and for the sake of brevity, they are not repeated here. In this way, the grayscale differences between the graph P and each of the translated graphs P1, P2, and P3 can reflect the graph P and the similarity degree with each of the translated graphs P1, P2, and P3.

[0024] Next, in step S222, the processor 12 selects a minimum grayscale difference among the grayscale differences corresponding to the translated graphs P1, P2, and P3 (for example: the grayscale difference between the graph P and the translated graph P3), and determines whether the minimum grayscale difference is less than a threshold value. If the minimum grayscale difference is not less than the threshold value, the processor 12 determines that the similarity degree between the graph P and the translated graph is too low and executes step S223 to ignore marking the block where the graph P is located as the first-cycle image block. On the contrary, if the minimum difference is less than the grayscale threshold value, the processor 12 determines that there is a certain similarity degree and a periodic feature between the graph P and the translated graph. Therefore, the processor 12 can execute step S224 to mark the block where the graph P is located as the first-cycle image block, where the translation amount corresponding to the minimum grayscale difference is the graph change period of the graph P. In other words, assuming that there is a minimum grayscale difference between the graph P and the translated graph P3, it means that the grayscale change of the graph P is periodic with every 3M pixels.

[0025] Furthermore, in some embodiments, step S22 further includes steps S225 and S226. Please refer to Figure 8 the flowchart shown, in Figure 8 the embodiment, in addition to being able to execute the aforementioned steps S221 to S224, the processor 12 can also execute steps S225 and S226 to further confirm the periodic feature of the pixel P. Figure 8 Steps S221 to S224 in Figure 6 are similar to the corresponding steps in

[0026] As Figure 8 shown, in step S225, the processor 12 calculates the interleaving frequency of each pixel in the graph P corresponding to the average grayscale value of the graph P. Please also refer to Figure 7 From Figure 7 it can be seen that the number of intersections between the curve representing the grayscale of the graph P and the straight line representing its average grayscale value MN is 10. And since each periodic image change of the graph P will cause the above curve and straight line to intersect 2 times, therefore, the processor 12 can calculate that the interleaving frequency of the graph P is 5, which means that there are 5 periodic changes in the graph P.

[0027] In one embodiment, the process by which the processor 12 calculates the interleaving count is as follows: (1) Represent pixel values greater than the average value in the pattern P as 1, and represent pixel values less than the average value in the pattern P as -1; (2) Perform an exclusive OR (XOR) operation on every two adjacent pixels; and (3) Sum up the results of all the exclusive OR operations to obtain the interleaving count.

[0028] For ease of explanation, the translation amount corresponding to the minimum grayscale difference will be referred to as “the most similar translation amount” hereinafter. For example, in the embodiment where M is 1, since there is a minimum grayscale difference between the pattern P and the translated pattern P3, the “most similar translation amount” is the translation amount of 3. Next, in step S226, the processor 12 can calculate the difference by multiplying the interleaving frequency by the most similar translation amount and then subtracting the total number of pixels in the pattern P, and compare the difference with a second threshold. If the difference is higher than or equal to the second threshold, the processor 12 will execute step S223. If the difference is lower than the second threshold, indicating that the pattern change period (i.e., the most similar translation amount) of the pattern P calculated through step S221 and the interleaving frequency calculated through step S225 match each other, the processor 12 will execute step S224.

[0029] For example, in the embodiment where M is 1, the translation amount corresponding to the aforementioned minimum grayscale difference for pixel P3 is 3 (i.e., the most similar translation amount), and the interleaving frequency calculated in step S225 is 5. Assume that there are 16 pixels in pixel P and the second threshold is 3. Then the processor 12 can calculate that the absolute value of the product of the most similar translation amount and the interleaving frequency (i.e., 15) minus the number of pixels is 1, and use the calculated absolute value after subtraction as the difference. Since the difference is less than the second threshold, the processor 12 can execute step S224 to mark the block where the pattern P is located as the first periodic image block. In contrast, if the difference calculated by the processor 12 through the aforementioned operations is not less than the second threshold, the processor 12 will execute step S223 and ignore marking the block where the pattern P is located as the first periodic image block.

[0030] In summary, in Figure 8 the embodiment, the processor 12 can improve the accuracy of determining whether the pattern P has periodic characteristics through the additional steps S225 and S226. It should be noted that Figure 8 the order of the steps S221, S222, S225, and S226 shown is one example of implementation, and the technology of the present application is not limited thereto. In fact, the processor 12 can execute these steps in other orders. For example, the processor 12 can execute steps S221 and S225 simultaneously, and execute steps S222 and S226 simultaneously, and then select to execute step S223 or S224 according to the results of steps S222 and S226.

[0031] It should be noted that in the above-mentioned multiple embodiments, although the graphic P is translated to the right to generate the translated graphics P1, P2, and P3, the translation direction of the graphic P and the number of translated graphics are not limited to the above. In other embodiments, the image processing device 1 can also perform similar comparison operations after translating the graphic P in different directions to detect periodic images distributed in different directions, which will not be elaborated here for the sake of simplicity.

[0032] Please return to Figure 3 , in step S23, the processor 12 of the image processing device 1 performs a first motion estimation 1ME on the image F1 and the image Ff1 based on the at least one first marker to generate a motion vector MV1.

[0033] Specifically, when the processor 12 performs the first motion estimation 1ME and generates the motion vector MV1, it adjusts the motion vectors MV1 corresponding to the blocks with the first marker in the image F1 (i.e., the first periodic image blocks) to avoid the subsequent compensation image from being fragmented.

[0034] In some embodiments, please refer to Figure 9 , the multiple scanning operations in the first motion estimation 1ME in step S23 further include steps S231 to S234, and the processor 12 can adjust the search window corresponding to the first periodic image blocks by executing steps S231 to S234.

[0035] In step S231, the processor 12 generates a search window corresponding to each of these first blocks in the image F1. In step S232, the processor 12 adjusts the corresponding search window based on an extension direction (e.g., Figure 7 the graphic P extends to the right) of the periodic features in each of the first periodic image blocks marked in step S22 and the most similar translation amount. In step S233, the processor 12 generates a plurality of candidate vectors within the search window. In step S234, the processor 12 selects the first motion vector MV1 corresponding to the first periodic image block from these candidate vectors.

[0036] Specifically, when the processor 12 performs the scanning operation of the first motion estimation 1ME, A search window for generating a first periodic image block is generated in the extending direction of the periodic feature. Then, the search window may overlap the periodic image of the first periodic image block, which may cause the first motion estimation (1ME) to generate an incorrect motion vector and ultimately result in image fragmentation. Therefore, in step S232, the processor 12 can avoid generating a search window in the extending direction of the periodic feature of each first periodic image block. Also, in step S232, the processor 12 can further avoid setting the size of the search window to a multiple of the most similar translation amount (i.e., the graphic change period of the periodic feature). In some embodiments, step S232 can be omitted.

[0037] Next, as Figure 4 shown, after completing the first motion estimation (1ME) (i.e., after completing step S23), the processor 12 can then continue to mark the blocks with periodic features for each of the other-sized images F2 to Fn, and then perform motion estimation. Since the processor 12 performs similar operations on each of the images F2 to Fn, for the sake of simplicity, this application only uses steps S24 and S25 to illustrate the operations performed by the processor 12 on the image Fn.

[0038] Please go back to Figure 3 , in step S24, the processor 12 of the image processing device 1 marks at least one of the multiple nth blocks in the image FN with at least one nth mark as at least one nth periodic image block through an operation similar to step S22, where the at least one nth periodic image block has an nth periodic feature, and n can be a positive integer not less than 2.

[0039] It should be noted that for images F2 to Fn of different sizes (resolutions), when determining whether each block in the image has a periodic feature, the graphic P for comparison can be formed by the same number of pixels.

[0040] Since images with smaller sizes (e.g., images F1, F2) have lower resolutions, their graphic P can be used to identify periodic features with a wider distribution range in the image Fn. In contrast, since images with larger sizes (e.g., images Fn-1, Fn) have higher resolutions, their graphic P can be used to identify periodic features with a narrower distribution range in the image Fn. In this way, through a hierarchical marking operation, the processor 12 can generate the first to nth marks corresponding to periodic features with different distribution ranges.

[0041] In some embodiments, each time a mark is generated, the processor 12 combines the previously generated marks to retain the blocks determined to have periodic images in the previous motion estimation. "Merge" can be understood as a logical OR operation. In other words, after the processor 12 generates one or more second tags, it will perform a logical OR operation on the second tags and the corresponding one or more first tags respectively to update the second tags; after the processor 12 generates one or more nth tags, it will perform a logical OR operation on the nth tags and the corresponding one or more (n - 1)th tags respectively to update the nth tags, and so on.

[0042] Further, in step S25, the processor 12 of the image processing device 1 performs an nth motion estimation nME on the image Fn and the image Ffn based on a plurality of motion vectors MVn - 1 and the at least one nth tag, where n can be a positive integer not less than 2.

[0043] Similar to step S23, the processor in step S25 can generate the motion vector MVn through the same operation. For the search operation of adjusting the corresponding search window for the nth cycle image block in step S25, please refer to Figure 10 , step S25 may include steps S251 to S254, where step S251 corresponds to step S231, step S252 corresponds to step S232, step S253 corresponds to step S233, and step S254 corresponds to step S234. In some embodiments, similar to step S232, step S252 may be omitted.

[0044] However, different from step S23, when the processor 12 performs the second to nth motion estimations 2ME to nME, it will refer to the previously generated motion vectors. For example, the motion vector corresponding to the same block generated in the previous motion estimation is used as the initial vector in the current motion estimation to accelerate the convergence speed when scanning the motion vectors.

[0045] In some embodiments, when the processor 12 selects the motion vector corresponding to the cycle image block from the candidate vectors, it may also refer to the motion vectors generated in the previous motion estimation. Please refer to Figure 11 , which are steps S2531 to S2534 included in step S253.

[0046] In step S2531, in response to one of the candidate vectors corresponding to the at least one nth cycle image block, the processor 12 calculates a difference between one of the plurality of reference motion vectors corresponding to the one of the at least one nth cycle image block and each of the candidate vectors.

[0047] In step S2532, the processor 12 calculates a penalty value for each of the candidate vectors based on the difference corresponding to each of the candidate vectors, where the difference and the penalty value are positively correlated.

[0048] In step S2533, the processor 12 reduces the weight of each of these candidate vectors based on the penalty value of each of these candidate vectors.

[0049] In step S2534, the processor 12 selects one of these nth motion vectors MVn from these candidate vectors based on the weight of each of these candidate vectors.

[0050] Specifically, when the processor 12 selects a motion vector corresponding to the nth cycle image block, it first performs step S2531 to calculate the difference between the candidate vector and the reference motion vector (for example: the absolute value after subtracting the two vectors).

[0051] In some embodiments, the reference motion vector is the motion vector corresponding to the same block in the previous motion estimation (for example: the n - 1th motion estimation). In this way, the result of the previous motion estimation can be referred to select the motion vector generated by the current motion estimation.

[0052] In some embodiments, the reference motion vector is the regional motion vector corresponding to the same block in the previous motion estimation (for example: the n - 1th motion estimation). The regional motion vector can be obtained by averaging the sum of the motion vectors corresponding to this block and other cycle image blocks with similar positions. In this way, the motion vector corresponding to the block with periodic characteristics in adjacent blocks in the previous motion estimation can be referred to select the motion vector generated by the current motion estimation.

[0053] Next, the larger the difference corresponding to the candidate vector, the greater the difference between the candidate vector and the result of the previous motion estimation. Then, when the processor 12 executes step S2532, a higher penalty value will be given; conversely, the smaller the difference corresponding to the candidate vector, the smaller the difference between the candidate vector and the result of the previous motion estimation. Then, when the processor 12 executes step S2532, a smaller penalty value will be given.

[0054] Then, the processor 12 executes step S2533, adjusts the weight of the candidate vector based on the penalty value. The higher the penalty value, the greater the reduction in the weight. Conversely, the lower the penalty value, the smaller the reduction in the weight.

[0055] Finally, the processor 12 executes S2534, and selects the nth motion vector MVn from the candidate vectors based on the adjusted weight.

[0056] In this way, the processor 12 can adjust the weight of the candidate vector according to the difference between the candidate vector and the motion vector generated by the previous motion estimation. In this way, the result of the previous motion estimation can be introduced as a judgment factor at the same time. On the other hand, if a certain candidate vector has a relatively high weight compared to other candidate vectors, even if there is a certain difference from the motion vector generated by the previous motion estimation, it still has a chance to be selected as the motion vector and will not be directly excluded.

[0057] In some embodiments, when the processor 12 selects the motion vector corresponding to the periodic image block from the candidate vectors, it can also directly exclude some candidate vectors and select the motion vector from the remaining other candidate vectors.

[0058] For example, the spatial candidate vector is a candidate vector generated after referring to the motion vectors of adjacent blocks. When the block is marked as a periodic image block, the reference of the motion vectors of adjacent blocks is relatively low. Therefore, the processor 12 can exclude the spatial candidate vectors from the candidate vectors.

[0059] In another example, the temporal candidate vector is a candidate vector generated after referring to the motion vectors of the previous time frame. When the block is marked as a periodic image block, the reference of the motion vectors of the previous time frame is relatively low. Therefore, the processor 12 can exclude the temporal candidate vectors from the candidate vectors.

[0060] In yet another example, the random candidate vector is a randomly generated candidate vector. When the block is marked as a periodic image block, the risk of generating fragmented images by using the random candidate vector as the motion vector is relatively high. Therefore, the processor 12 can exclude the random candidate vectors from the candidate vectors.

[0061] Finally, in step S26, the processor 12 of the image processing device 1 performs a motion compensation on the image Fn and the image Ffn based on the motion vector MVn generated by the nth motion estimation to generate a compensated frame image between the image Fn and the image Ffn.

[0062] In summary, the image processing apparatus 1 provided in the present application can provide the motion estimation of a larger-size image as a reference based on the motion estimation result of a smaller-size image. Before the motion vectors are generated in the motion estimation operation, the image processing apparatus 1 can also detect whether there are periodic images in the image, and then adjust the output of the motion estimation. In addition, through the detection operations for images of different sizes, the image processing apparatus 1 can detect periodic images in different range ratios. When detecting periodic images, the image processing apparatus 1 can confirm whether it conforms to the periodic characteristics based on two aspects: translational comparison and calculation of the pixel change frequency. When generating candidate vectors, the image processing apparatus 1 can adjust the search window with reference to the periodic characteristics to avoid searching for incorrect blocks. When selecting motion vectors, the image processing apparatus 1 can adjust the weights of the candidate vectors for the blocks with periodic images to introduce the factors of the previous motion estimation. In this way, the image processing apparatus 1 can detect the positions where repetitive images appear in the image during the motion estimation and correct the motion estimation results, while taking into account periodic images of various sizes and ranges as well as the operation efficiency.

[0063] Although several embodiments are described in detail above as examples, the image processing apparatus and method proposed in the present application can also be implemented by other systems, hardware, software, storage media, or combinations thereof. Therefore, the protection scope of the present application should not be limited to the specific implementation manners described in the embodiments of the present application, but should be determined by the appended claims.

[0064] It is obvious to those of ordinary skill in the technical field to which the present application belongs that various modifications and changes can be made to the structure of the present application without departing from the scope or spirit of the present application. In view of the foregoing, the protection scope of the present application also covers the modifications and changes made within the scope of the appended claims.

Symbol Description

Claims

1. An image processing apparatus, characterized in that, comprising: a memory for storing a current image and a reference image; and a processor coupled to the memory for performing the following operations: scaling down the current image and the reference image to respectively generate a scaled-down current image and a scaled-down reference image; marking at least one of a plurality of first blocks in the scaled-down current image with at least one first mark as at least one first periodic image block, wherein the at least one first periodic image block has a first periodic characteristic; performing a first motion estimation on the scaled-down current image and the scaled-down reference image based on the at least one first mark to generate a plurality of first motion vectors; marking at least one of a plurality of nth blocks in the current image with at least one nth mark as at least one nth periodic image block, wherein the at least one nth periodic image block has an nth periodic characteristic; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the at least one nth mark to generate a plurality of nth motion vectors; and performing a motion compensation on the current image and the reference image based on the nth motion vectors to generate an interpolated image between the current image and the reference image.

2. The image processing apparatus according to claim 1, characterized in that, the operation of marking the at least one first periodic image block further comprises: comparing a plurality of pixels in the current image with a plurality of translated pixels to calculate a plurality of differences, wherein the translated pixels are the results of translating the pixels based on different translation amounts, and each of the differences corresponds to one of the translation amounts; and judging whether to mark the pixel as the at least one first periodic image block based on one of the minimum differences among the differences and a threshold value.

3. The image processing apparatus according to claim 2, characterized in that, the operation of marking the at least one first periodic image block further comprises: calculating an interleaving frequency of the pixel corresponding to an average value of the pixel; and judging whether to mark the pixel as the at least one first periodic image block based on one of the minimum differences among the differences, a threshold value, the interleaving frequency, and a most similar translation amount corresponding to the minimum difference.

4. The image processing apparatus according to claim 2, characterized in that, the first motion estimation includes a plurality of scanning operations, and the scanning operations further comprise: generating a search window corresponding to each of the first blocks in the scaled-down current image; adjusting the corresponding search window based on a translation direction of each of the at least one first periodic image block and the most similar translation amount; generating a plurality of candidate vectors within the search window; and selecting the first motion vector corresponding to each of the at least one first periodic image block from the candidate vectors.

5. The image processing apparatus according to claim 1, characterized in that, the nth motion estimation includes a plurality of scanning operations, and the scanning operations further comprise: generating a search window corresponding to each of the nth blocks in the scaled-down current image; generating a plurality of candidate vectors within the search window; and Select the nth motion vector corresponding to each of the at least one nth-period image block from the candidate vectors.

6. The image processing apparatus according to claim 5, wherein, the operation of selecting one of the nth motion vectors further includes: in response to the candidate vector corresponding to one of the at least one nth-period image block, calculating a difference between one of the reference motion vectors corresponding to that one of the at least one nth-period image block and each of the candidate vectors; calculating a penalty value for each of the candidate vectors based on the difference corresponding to each of the candidate vectors, wherein the difference and the penalty value are positively correlated; reducing the weight of each of the candidate vectors based on the penalty value of each of the candidate vectors; selecting one of the nth motion vectors from the candidate vectors based on the weight of each of the candidate vectors.

7. The image processing apparatus according to claim 6, wherein, the reference motion vector is one of the (n - 1)th motion vectors corresponding to that one of the at least one nth-period image block.

8. The image processing apparatus according to claim 6, wherein, the reference motion vector is a regional motion vector corresponding to that one of the at least one nth-period image block, wherein the regional motion vector corresponds to that one of the at least one nth-period image block and at least one period image block adjacent in position to that one of the at least one nth-period image block.

9. The image processing apparatus according to claim 1, wherein, marking at least one of the nth blocks in the current image as at least one nth-period image block with the at least one nth mark includes: performing a logical OR operation on the at least one nth mark and the at least one first mark respectively to update the at least one nth mark.

10. An image processing method, applicable to an electronic device, wherein, its steps include: shrinking a current image and a reference image respectively to generate a shrunk current image and a shrunk reference image; marking at least one of a plurality of first blocks in the shrunk current image as at least one first-period image block with at least one first mark, wherein the at least one first-period image block has a first periodic feature; performing a first motion estimation on the shrunk current image and the shrunk reference image based on the at least one first mark to generate a plurality of first motion vectors; marking at least one of a plurality of nth blocks in the current image as at least one nth-period image block with at least one nth mark, wherein the at least one nth-period image block has an nth periodic feature; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the at least one nth mark to generate a plurality of nth motion vectors; and performing a motion compensation on the current image and the reference image based on the nth motion vectors to generate an interpolated frame image between the current image and the reference image.

Images