Image processing apparatus and method

Abstract

The invention discloses an image processing device and method. The image processing device is used for performing the following operations: performing a first motion estimation on a reduced current image and a reduced reference image to generate a plurality of first motion vectors and a first reliability corresponding to the first motion vectors; an nth motion estimation is performed on a current image and a reference image based on the first motion vectors and the first reliability to generate a plurality of nth motion vectors and an nth reliability corresponding to the nth motion vectors. And generating a complementary frame image between the current image and the reference image based on the nth motion vectors and the nth reliability.

Description

Technical Field

[0001] The present application relates to an image processing apparatus and method, and particularly to an image processing apparatus and method for motion estimation (ME) and motion compensation (MC). Background Art

[0002] In the field of image processing, when performing frame rate conversion (FRC), a motion vector can be calculated through motion estimation, and after processing, it is handed over to motion compensation to generate an interpolated image between two original images, so that the image is smoother.

[0003] However, when performing motion compensation, there is a lack of an evaluation mechanism for motion vectors to measure the reliability of motion vectors. Therefore, when the reliability of motion vectors is too low, misjudgment will occur during motion compensation, resulting in broken, edge shaking, or judder in the image.

[0004] In view of this, for the estimation of motion vectors, how to balance the computational cost and accuracy is an urgent goal for the industry to strive for. Summary of the Invention

[0005] To solve the above problems, the present application provides an image processing apparatus, including a memory and a processor. The memory is used to store a current image and a reference image. The processor is coupled to the memory and is configured to perform the following operations: downscale the current image and the reference image to respectively generate a downscaled current image and a downscaled reference image; perform a first motion estimation on the downscaled current image and the downscaled reference image to generate a plurality of first motion vectors and a first reliability corresponding to the first motion vectors; perform an nth motion estimation on the current image and the reference image based on the first motion vectors and the first reliability to generate a plurality of nth motion vectors and an nth reliability corresponding to the nth motion vectors; and perform a motion compensation on the current image and the reference image based on the nth motion vectors and the nth reliability to generate an interpolated image between the current image and the reference image.

[0006] The present application also provides an image processing method applicable to an electronic device, including: shrinking a current image and a reference image to respectively generate a shrunk current image and a shrunk reference image; performing a first motion estimation on the shrunk current image and the shrunk reference image to generate a plurality of first motion vectors and a first reliability corresponding to the first motion vectors; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the first reliability to generate a plurality of nth motion vectors and an nth reliability corresponding to the nth motion vectors; and generating an interpolated image between the current image and the reference image based on the nth motion vectors and the nth reliability.

[0007] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory, and are intended to provide further explanation of the present application as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To make the above and other objects, features, and advantages of the present application more apparent and understandable, the description of the accompanying drawings is as follows: Figure 1 A schematic diagram for generating an interpolated image between two images by motion estimation and motion compensation operations; Figure 2 A schematic diagram of an image processing device in an embodiment of the present application; Figure 3 A schematic diagram of a motion estimation operation in an embodiment of the present application; Figure 4 A schematic diagram of shrinking an image in an embodiment of the present application; and Figure 5 A flowchart of an image processing method in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0009] To make the description of the present application more detailed and complete, reference may be made to the accompanying drawings and the following various embodiments, where the same reference numerals in the drawings represent the same or similar components.

[0010] Motion estimation and motion compensation are used to generate a compensating image between two images to increase the frame rate. For example, please refer to Figure 1 , image Fk-1 and image Fk are two adjacent frames in a video. Motion estimation operation and motion compensation operation are used to generate interpolated images FC1 to FC4 between image Fk-1 and image Fk. When performing the motion estimation operation, image Fk-1 and image Fk can be divided into i by j blocks, and the best motion vector of each block can be found by using 3D recursive search.

[0011] More specifically, the three-dimensional recursive search includes multiple scan operations. Each scan operation generates candidates for various motion vectors within the search window for each block of the entire image, based on the initial vector of the block and the characteristics of various image changes. For example, candidates such as zero, spatial, temporal, global, etc. candidate vectors are generated, and the one with the highest matching degree is calculated as the motion vector of this block. Then, when the scan operation is performed again, the scan operation adds a random vector to the motion vector of each block obtained in the previous scan as the initial vector of this block. Thus, through multiple scans, the best motion vector for each block (such as the motion vectors MV1 to MVn described later) can be converged.

[0012] After that, the motion compensation operation generates interpolated images FC1 to FC4 between image Fk-1 and image Fk based on image Fk-1, image Fk, and the best motion vector of each block. For example, if the best motion vector is roughly a vector from the lower left to the upper right, then according to the circle in the lower left corner of image Fk-1 and the circle in the upper right corner of image Fk, the multiple circles in interpolated images FC1 to FC4 will be arranged in sequence from the lower left to the upper right. In other words, through the motion estimation operation and the motion compensation operation, interpolated images FC1 to FC4 can be generated based on image Fk-1 and image Fk.

[0013] Furthermore, the motion estimation operation can generate a reliability level of the motion vector based on one or more of the following information regarding the motion vector: Sum of Absolute Differences (SAD), average luminance of the image, regional motion vector, and global motion vector. After calculating the reliability, the result of the motion compensation operation can be adaptively adjusted based on the reliability (for example: adjusting the positions of interpolated images FC1 to FC4 on the time axis).

[0014] In some embodiments, after the motion estimation operation ends, the reliability of the entire image (e.g., image Fk-1) is calculated once, and the reliability calculated once is further used for the motion compensation operation. However, due to the limitation of memory space, the reliability calculated once may not provide sufficient information for the motion compensation operation, resulting in inaccurate compensation. Therefore, it is necessary to increase other compensation methods or expand the memory space to make up for the lack of compensation accuracy. For image processing technologies that emphasize computational cost and efficiency, such a calculation method is likely to result in too high computational cost or low computational efficiency.

[0015] Therefore, the present application proposes an image processing device. Please refer to Figure 2 , which is a schematic diagram of the image processing device 1 in an embodiment of the present application. As Figure 2 shown, the image processing device 1 includes a processor 12 and a memory 14, wherein the processor 12 is coupled to the memory 14.

[0016] In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit, a multi-processor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable arithmetic unit.

[0017] The memory 14 is used to store the current image Fn and the reference image Ffn. In some embodiments, the memory 14 may include a semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk, and / or an optical disk.

[0018] First, the processor 12 of the image processing device 1 reduces the current image Fn and the reference image Ffn to respectively generate a plurality of reduced current images F1 to Fn-1 with different resolutions and a plurality of reduced reference images Ff1 to Ffn-1 with different resolutions.

[0019] Specifically, please refer to Figure 3 and Figure 4 , Figure 3 , which is a schematic diagram of the motion estimation operation MEP in an embodiment of the present application. Figure 4This is a schematic diagram of reducing an image in an embodiment of the present application. Images Fn-1 and Ffn-1 are the images obtained by reducing the current image Fn and the reference image Ffn by one size reduction factor, respectively. By analogy, images F1 and Ff1 are the images obtained after n-1 size reduction factors. For example, the size reduction factor can be 1 / 2 or 1 / 4, but the present application is not limited thereto. In one embodiment, the current image Fn is Figure 1 the image Fk-1 of Figure 1 and the reference image Ffn is the image Fk of

[0020] That is, the reference image Ffn and the current image Fn can be two adjacent frames in a continuous image (e.g., a video), but the present application is not limited thereto. The motion estimation operation MEP is a hierarchical operation, that is, the motion estimation operation MEP includes the first motion estimation 1ME to the nth motion estimation nME executed in sequence. The processor 12 can perform the first motion estimation 1ME on the images F1 and Ff1 to generate a plurality of first motion vectors MV1 and a first reliability RL1. Specifically, after calculating the plurality of first motion vectors MV1, the image processing device 1 can also calculate the first reliability RL1 corresponding to the plurality of first motion vectors MV1 based on the relevant information of the plurality of first motion vectors MV1.

[0021] In some embodiments, the processor 12 can calculate the first reliability RL1 based on the plurality of first motion vectors MV1 and the sum of absolute differences corresponding to the plurality of first motion vectors MV1, where the sum of absolute differences is positively correlated with the first reliability RL1. For example, the image processing device 1 can calculate the reliability corresponding to the plurality of motion vectors based on the following Equation (1). [Equation (1)] RL = ∑(Mv speed - base threshold) * (SAD value - coring threshold) * RMV relation * GMV relation

[0022] where RL is the reliability (e.g., the first reliability RL1), MV speed is the magnitude of each motion vector (e.g., the first motion vector MV1), base threshold is the reference threshold, SAD value is the sum of absolute differences, coring threshold is the core threshold, RMV relation is the correlation degree between the plurality of motion vectors and the regional motion vector, GMV relation is the correlation degree between the plurality of motion vectors and the global motion vector, and ∑ is to add the plurality of products calculated based on the plurality of motion vectors.

[0023] It should be noted that the regional motion vector is the motion vector calculated based on the pixels in the region composed of multiple adjacent blocks in the image during motion estimation; the global motion vector is the motion vector calculated based on the pixels in the entire image during motion estimation.

[0024] Furthermore, the aforementioned GMV relation can be calculated by one of the following three calculation methods shown in Equation 2. [Equation 2] GMV relation =(MV - base_th1).clip(min_value, max_value)+(MV - base_th2).clip(min_value, max_value) / / 2 GMV relation =max((MV - base_th1).clip(min_value, max_value),(MV - base_th2).clip(min_value, max_value)) GMV relation =min((MV - base_th1).clip(min_value, max_value),(MV - base_th2).clip(min_value, max_value))

[0025] Where MV is the motion vector (for example, one of the first motion vectors MV1), base_th1 and base_th2 are the global motion vector magnitudes with the highest weight and the second highest weight respectively (that is, the global motion vector magnitudes with the highest credibility and the second highest credibility). The clip function is used to control the difference between the calculated motion vector and the global motion vector between the preset minimum value (that is, min_value) and the preset maximum value (that is, max_value). For example, if the difference is less than the preset minimum value, the difference is set to the preset minimum value; if the difference is greater than the preset maximum value, the difference is set to the preset maximum value. / / is the floor division operator (that is, after dividing, the decimal part of the resulting quotient is unconditionally discarded to obtain an integer not greater than the quotient), the max function is to take the maximum value, and the min function is to take the minimum value. It can be seen from this that the GMV relation can be obtained by calculating the average, maximum or minimum value of the difference between the motion vector and the global motion vector with a higher weight.

[0026] As for base_th1 and base_th2, they can be calculated by the following Equation 3. [Equation Three] base_th1 = sum(abs(gmv_1st__mvx + gmv_1st_mvy)) base_th2 = sum(abs(gmv_2nd_mvx + gmv_2nd_mvy))

[0027] Where gmv_1st_mvx and gmv_2nd_mvx are the x-axis components of the global motion vectors with the highest and the second highest weights respectively, and gmv_1st_mvy and gmv_2nd_mvy are the y-axis components of the global motion vectors with the highest and the second highest weights respectively. The abs function is used to calculate the absolute value, and the sum function is used to calculate the total sum.

[0028] On the other hand, the aforementioned RMV relation can also be calculated by a calculation method similar to that of the GMV relation. The only difference is that when calculating the RMV relation, the motion vectors in the region where the current calculation block is located are referred to, while when calculating the GMV relation, all the motion vectors in the entire image range are referred to.

[0029] As can be seen from Equation One, the reliability is positively correlated with the magnitude of the motion vector, the sum of absolute errors, the degree of correlation between the motion vector and the regional motion vector, and the degree of correlation between the motion vector and the global motion vector. In other words, when the motion vector is larger, the sum of absolute errors is larger, the degree of correlation between the motion vector and the regional motion vector is higher, and / or the degree of correlation between the motion vector and the global motion vector is higher, the reliability of the motion vector is higher.

[0030] In some embodiments, the reliability can be represented by 8 bits, that is, an integer between 0 and 255. The higher the value, the higher the reliability of the corresponding motion vector, and the lower the value, the lower the reliability of the corresponding motion vector.

[0031] Next, the processor 12 of the image processing device 1 performs a second motion estimation 2ME on the image F2 and the image Ff2 based on a plurality of first motion vectors MV1 and first reliabilities RL1 previously generated from the smaller-sized images F1 and Ff1, so as to generate a plurality of second motion vectors MV2 and second reliabilities RL2 corresponding to the plurality of second motion vectors MV2. For example: The second motion estimation 2ME can use the product of a certain first motion vector MV1 and the first reliability RL1 as the initial vector of a corresponding block in the scanned image F2 or the image Ff2. In this way, the second motion estimation 2ME can refer to the result of the first motion estimation 1ME to accelerate the convergence rate of the plurality of second motion vectors MV2.

[0032] In some embodiments, the operation of the processor 12 to generate the second reliability RL2 further includes generating an intermediate reliability based on a plurality of second motion vectors MV2; and generating the second reliability RL2 based on the first reliability RL1 and the intermediate reliability, where the first reliability RL1 and the second reliability RL2 are positively correlated, and the intermediate reliability and the second reliability RL2 are positively correlated.

[0033] For example, the processor 12 generates the intermediate reliability based on a plurality of second motion vectors MV2 and related information in a manner similar to the operation of generating the first reliability RL1 in the foregoing embodiments, and then adds the intermediate reliability and the first reliability RL1 as the second reliability RL2, or takes the average value of the intermediate reliability and the first reliability RL1 as the second reliability RL2.

[0034] In this way, the processor 12 can reflect the first reliability RL1 generated by the first motion estimation 1ME in the second reliability RL2 generated by the second motion estimation 2ME.

[0035] In some embodiments, the operation of the processor 12 to perform the second motion estimation 2ME further includes calculating the degrees of matching of a plurality of candidate vectors (for example: zero, spatial, temporal, global, etc. candidate vectors); and using one of the plurality of candidate vectors corresponding to the highest of these degrees of matching as one of the plurality of second motion vectors MV2. The operation of calculating these degrees of matching further includes: calculating the vector difference between one of these candidate vectors and the plurality of first motion vectors MV1 corresponding to the candidate vector; calculating a penalty value based on the vector difference and the first reliability RL1, where the penalty value is positively correlated with the first reliability RL1 and the vector difference; and reducing the degrees of matching of these candidate vectors based on the penalty value.

[0036] For example, the processor 12 can subtract these candidate vectors from the corresponding plurality of first motion vectors MV1 respectively to obtain the vector differences, and then multiply the vector differences by the first reliability RL1 to obtain the penalty values corresponding to each candidate vector. In summary, the penalty value will reflect the difference between the plurality of candidate vectors and the corresponding plurality of first motion vectors MV1, and will also reflect the first reliability RL1 of the plurality of first motion vectors MV1.

[0037] Further, the processor 12 reduces the matching degree of each candidate vector based on the penalty value of the candidate vector. For example, if the absolute value of the vector difference between multiple candidate vectors and multiple first motion vectors MV1 is large, indicating that these candidate vectors do not match the result of the first motion estimation 1ME well, the processor 12 will significantly reduce the matching degree according to the higher penalty value. On the other hand, if the reliability corresponding to multiple first motion vectors MV1 is high, indicating that the result of the first motion estimation 1ME is highly reliable, the processor 12 will also significantly reduce the matching degree according to the higher penalty value. In short, the processor 12 reduces the matching degree of these candidate vectors based on the absolute value of the vector difference and the reliability to reflect the result of the first motion estimation 1ME.

[0038] In some embodiments, similar to the operation of the aforementioned three-dimensional recursive search, the second motion estimation 2ME includes multiple scanning operations, and these scanning operations further include: the processor 12 calculates a search window size of the second motion estimation 2ME based on the first reliability RL1; and the processor 12 generates multiple candidate vectors within a search window having the search window size.

[0039] Specifically, the processor 12 can adjust the range of searching for multiple candidate vectors in the image during scanning in the three-dimensional recursive search based on the reliability. In some embodiments, the processor 12 can obtain the search window size corresponding to the reliability based on a look-up table, where the look-up table can be stored in the memory 14 and records the search window sizes corresponding to different reliability values. For example, when the reliability is a value between 0 and 32, the search window size is 5*5 pixels.

[0040] In some embodiments, the processor 12 can calculate the search window size of the second motion estimation 2ME based on the reliability (e.g., the first reliability RL1) using the following Equation 4. [Equation 4] search step=(original search step*reliability level)>>reliabilitylevel bit number

[0041] Where search step is the search window size, original search step is the default search window size (e.g., 3*3 pixels), reliability level is the reliability (e.g., the first reliability RL1), reliabilitylevel bit number is the number of bits of the reliability (e.g., 8), and the symbol ">>" represents a right shift of bits.

[0042] According to Equation Four above, the processor 12 can adjust the degree of reduction of the search window size based on the reliability. When the reliability is higher, the search window size will be closer to the default search window size and less likely to be reduced. Conversely, when the reliability is lower, the degree of reduction of the search window size will be greater.

[0043] In some embodiments, the processor 12 can also calculate the search window size based on the reliability (e.g., the first reliability RL1) according to Equation Five below. [Equation Five] search step=(original search step*(max reliability level-reliabilitylevel))>>reliability level bit number

[0044] Where search step is the search window size, original search step is the default search window size (e.g., 3*3 pixels), max reliability level is the maximum value of the reliability (e.g., 255), reliabilitylevel is the reliability (e.g., the first reliability RL1), and reliability level bit number is the number of bits of the reliability (e.g., 8).

[0045] According to Equation Five above, the processor 12 can adjust the degree of reduction of the search window size based on the reliability. When the reliability is lower, the search window size will be closer to the default search window size and will not decrease. Conversely, when the reliability is higher, the degree of reduction of the search window size will be greater.

[0046] In this way, the processor 12 can adjust the search window size based on the reliability, and then adjust the convergence speed of the motion vector based on the reliability generated during the previous motion estimation operation during the motion estimation operation. In addition, in the above embodiments where the search window size is calculated according to Equation Four and Equation Five, there is a linear correlation between the search window size and the reliability.

[0047] Next, the processor 12 performs a third motion estimation 3ME on the image F3 and the image Ff3 based on the multiple second motion vectors MV2 and the second reliability RL2 generated by the second motion estimation 2ME. The calculation process of the third motion estimation 3ME can be the same as that of the second motion estimation 2ME, except that the data used in the calculation process is different (for example, the image F3, the image Ff3, the multiple second motion vectors MV2, and the second reliability RL2 are used in the calculation process of the third motion estimation 3ME). By analogy, the processor 12 can perform motion estimation on the images F1 to Fn and the images Ff1 to Ffn of different sizes in the same way, and after performing the nth motion estimation, generate multiple nth motion vectors MVn and nth reliability RLn corresponding to the current image Fn and the reference image Ffn.

[0048] It should be noted that n can be a positive integer greater than 1. The image processing device 1 can adjust the number of times of reducing the current image Fn and the reference image Ffn, and the number of levels of the motion estimation operation MEP according to the consideration of computing performance and cost.

[0049] After the processor 12 completes the motion estimation operation MEP on the current image Fn and the reference image Ffn, the processor 12 of the image processing device 1 is used to perform a motion compensation operation (not shown) on the current image Fn and the reference image Ffn based on the multiple nth motion vectors MVn and the nth reliability RLn to generate one or more compensated frame images (not shown) between the current image Fn and the reference image Ffn. When the processor 12 performs motion compensation, it will refer to the nth reliability RLn to adjust the intervention degree of the multiple nth motion vectors MVn. In other words, when the processor 12 generates a compensated frame image based on the multiple nth motion vectors MVn, it will adjust the weights of these nth motion vectors MVn according to the nth reliability RLn. If the nth reliability RLn is higher, the weight of generating a compensated frame image by referring to these nth motion vectors MVn is higher; conversely, if the nth reliability RLn is lower, the weight of generating a compensated frame image by referring to these nth motion vectors MVn is lower.

[0050] In some embodiments, please refer to Figure 1 , the compensation targets of the compensated frame images FC1 to FC2 are the image Fk-1. Therefore, the time distances Din1 and Din2 from the compensated frame images FC1 to FC2 to the image Fk-1 are less than the time distances from the compensated frame images FC1 to FC2 to the image Fk. The compensation targets of the compensated frame images FC3 to FC4 are the image Fk. Therefore, the time distances Din3 and Din4 from the compensated frame images FC3 to FC4 to the image Fk are less than the time distances from the compensated frame images FC3 to FC4 to the image Fk-1.

[0051] Based on a similar concept, please refer to again Figure 2, when performing motion compensation, the processor 12 can select the image Ffn or Fn as the compensation target and use the following Equation Six to calculate one or more corresponding compensated frame images. [Equation Six] motion compensation output =(z data*(max reliability level - RLn)+mc data*reliability level)>>reliability level bit number

[0052] Where motion compensation output is the compensated frame image, z data is the compensation target (such as the image Fn), max reliability level is the maximum value of the reliability (for example: 255), mc data is the original compensated frame image that the processor 12 preliminarily calculates based on multiple motion vectors (such as multiple nth motion vectors MVn) and has not adjusted the image content based on the reliability, and reliability level bit number is the number of bits of the reliability (for example: 8).

[0053] The greater the difference between the maximum value of the reliability and the nth reliability RLn, the higher the visual similarity between the compensated frame image and the target image. The reason is that if the nth reliability RLn is lower, it means that the multiple nth motion vectors MVn are less reliable, so the compensated frame image can be made closer to the compensation target to reduce the compensation degree; conversely, if the nth reliability RLn is higher, it means that the multiple nth motion vectors MVn are more reliable, so the visual difference between the compensated frame image and the compensation target can be increased to increase the compensation degree.

[0054] In summary, the motion compensation operation includes: the processor 12 selects one of the current image Fn and the reference image Ffn as a compensation target; and generates a compensated frame image corresponding to the compensation target based on the compensation target, the nth reliability RLn, and multiple nth motion vectors MVn, where the nth reliability RLn is negatively correlated with a similarity between the compensated frame image and the compensation target.

[0055] In some embodiments, when the processor 12 performs the motion compensation operation, in addition to using the aforementioned Equation Six to determine the visual similarity between the compensated frame image and the target image, it will further use Equation Seven described later to determine the time distance between each compensated frame image and the compensation target according to the nth reliability RLn. Specifically, there will be an initial time distance between each compensated frame image calculated by the above Equation Six and its compensation target. The initial time distance is related to the total number of compensated frame images. For example, as Figure 1As shown, if it is expected to generate 4 supplementary frame images FC1 to FC4, the time distance between images Fk-1 and Fk can be divided into five equal parts to obtain the initial time distances Din1 to Din4 corresponding to the supplementary frame images FC1 to FC4 respectively (assuming that the time distances Din1 to Din4 are not adjusted after being generated during the motion compensation operation). Based on a similar concept, please refer to Figure 2 again. When the processor 12 performs the motion compensation operation, it can further correct the initial time distance of each supplementary frame image through the following formula VII to obtain the corrected time distance of each supplementary frame image. [Formula VII]

[0056] where original phase is the initial time distance, and phase boundary is the distance boundary of the supplementary frame image (for example: if it is expected to insert 128 supplementary frame images between the current image Fn and the reference image Ffn, then the 0th to 63rd supplementary frame images will correspond to the distance boundary 0 because they are closer to the current image Fn, and the 64th to 127th supplementary frame images will correspond to the distance boundary 128 because they are closer to the reference image Ffn). Adjust value is the correction value of the initial time distance. R is the maximum value of reliability (for example: 255).

[0057] The processor 12 will move the supplementary frame image towards the compensation target on the time axis according to the correction value to obtain the corrected time distance of the supplementary frame image. Continuing with the above example of inserting 128 supplementary frame images between the current image Fn and the reference image Ffn, when the processor 12 calculates that the initial time distance, reliability, and correction value of the supplementary frame image are 32, 64, and 8 respectively, since the compensation target of this supplementary frame image is the current image Fn (i.e., corresponding to the distance boundary 0), the corrected time distance of this supplementary frame image is the distance boundary plus the correction value, that is, 8.

[0058] In another example, when the original phase of the supplementary frame image generated by the processor 12 is 80 and the reliability is 128, the processor 12 can obtain a correction value of approximately 24 through the calculation of formula VII. Further, since the compensation target of the supplementary frame image is the reference image Ffn (i.e., corresponding to the distance boundary 128), the corrected time distance of the supplementary frame image is the distance boundary minus the correction value, that is, 104.

[0059] In summary, the processor 12 is used to calculate a correction value based on the initial time distance and the nth reliability RLn corresponding to the multiple nth motion vectors MVn, where the nth reliability RLn is negatively correlated with the correction value; and determine a corrected time distance between the supplementary frame image and the current image Fn or the reference image Ffn based on the correction value.

[0060] It can be seen from this that the processor 12 can adjust the insertion position of the frame-completed image between the current image Fn and the reference image Ffn based on the reliability. Equation 7 is used to adjust the insertion position of each frame-completed image according to the ratio between the nth reliability RLn and the maximum reliability. When the nth reliability RLn is higher, the correction value of the frame-completed image is smaller. On the contrary, when the nth reliability RLn is lower, the correction value of the frame-completed image is larger. The reason is that when the nth reliability RLn is higher, it means that the quality of the frame-completed image generated by multiple nth motion vectors MVn is higher, so a larger time distance can be set between the frame-completed image and the compensation target to improve the compensation degree; on the contrary, when the nth reliability RLn is lower, it means that the quality of the frame-completed image generated by multiple nth motion vectors MVn is lower, so the time distance between the frame-completed image and the compensation target can be reduced to lower the compensation degree.

[0061] In summary, the image processing device 1 provided in this application uses a hierarchical motion estimation operation MEP, in which each layer of motion estimation refers to the motion vectors and reliabilities generated by the previous layer of motion estimation. As a result, the image processing device 1 can adjust the intervention degree (e.g., weight) of the motion vectors in the next motion estimation based on the reliability, and can improve the efficiency and accuracy of calculating the motion vectors in each layer of motion estimation. In addition, the image processing device 1 also uses the nth reliability RLn generated by the motion estimation operation MEP as a reference for generating the frame-completed image in the motion compensation operation. Similarly, the image processing device 1 can adjust the intervention degree (e.g., weight) of the motion vectors in the motion compensation based on the reliability of the motion vectors. Therefore, while maintaining or reducing the computing cost, the image processing device 1 provided in this application can effectively reduce the occurrence of image fragmentation, fringing, or jerks.

[0062] This application also provides an image processing method. Please refer to Figure 5 , which is the flowchart of the image processing method 200 in the second embodiment of this application. The image processing method 200 is applicable to an electronic device (e.g., the image processing device 1). In some embodiments, the electronic device includes a memory (e.g., the memory 14) and a processor (e.g., the processor 12). The memory is used to store a current image and a reference image. The processor is coupled to the memory, and the processor is used to execute the image processing method 200.

[0063] The image processing method 200 includes steps S201 to S204. In step S201, the electronic device shrinks a current image and a reference image to respectively generate a shrunk current image and a shrunk reference image.

[0064] In step S202, the electronic device performs a first motion estimation on the downscaled current image and the downscaled reference image to generate a plurality of first motion vectors and a first reliability corresponding to these first motion vectors.

[0065] In step S203, the electronic device performs an nth motion estimation on the current image and the reference image based on these first motion vectors and the first reliability to generate a plurality of nth motion vectors and an nth reliability corresponding to these nth motion vectors.

[0066] In step S204, the electronic device generates an interpolation frame image between the current image and the reference image based on these nth motion vectors and the nth reliability.

[0067] In some embodiments, step S202 further includes the electronic device calculating the first reliability based on these first motion vectors and a plurality of sums of absolute differences corresponding to these first motion vectors, where these sums of absolute differences are positively correlated with the first reliability.

[0068] In some embodiments, step S202 further includes the electronic device calculating the first reliability based on a regional motion vector, a global motion vector, these first motion vectors, and these sums of absolute differences, where a first correlation degree between these first motion vectors and the regional motion vector is positively correlated with the first reliability, and a second correlation degree between these first motion vectors and the global motion vector is positively correlated with the first reliability.

[0069] In some embodiments, step S203 further includes the electronic device generating an intermediate reliability based on these nth motion vectors and the electronic device generating the nth reliability based on the first reliability and the intermediate reliability, where the first reliability is positively correlated with the nth reliability, and the intermediate reliability is positively correlated with the nth reliability.

[0070] In some embodiments, step S203 further includes the electronic device calculating a plurality of matching degrees of a plurality of candidate vectors; and using one of the candidate vectors corresponding to the highest of these matching degrees as one of these nth motion vectors.

[0071] In some embodiments, the step of calculating these matching degrees further includes the electronic device calculating a vector difference between one of these candidate vectors and one of these first motion vectors corresponding to this candidate vector; the electronic device calculating a penalty value based on the vector difference and the first reliability, where the penalty value is positively correlated with the vector difference and the first reliability; and the electronic device reducing one of these matching degrees of this candidate vector based on the penalty value.

[0072] In some embodiments, the nth motion estimation includes a plurality of scanning operations, and the scanning operations further include the electronic device calculating a search window size based on the first reliability, where there is a linear correlation between the first reliability and the search window size; and the electronic device generating a plurality of candidate vectors within a search window having the search window size.

[0073] In some embodiments, step S204 further includes the electronic device selecting one of the current image and the reference image as a compensation target; and the electronic device performing motion compensation based on the compensation target, the nth reliability, and the nth motion vectors to generate a compensated frame image corresponding to the compensation target, where the nth reliability is negatively correlated with a similarity between the compensated frame image and the compensation image.

[0074] In some embodiments, there is an initial time distance between the compensated frame image and the compensation target, and the image processing method further includes the electronic device calculating a correction value based on the initial time distance and the nth reliability, where the nth reliability and the correction value are negatively correlated; and the electronic device adjusting the initial time distance based on the correction value to determine a corrected time distance between the compensated frame image and the compensation target.

[0075] In some embodiments, determining the corrected time distance includes the electronic device moving the compensated frame image towards the compensation target on a time axis.

[0076] In some embodiments, the image processing method 200 further includes the electronic device shrinking the shrunk current image and the shrunk reference image to respectively generate a further shrunk current image and a further shrunk reference image; and the electronic device performing a third motion estimation on the further shrunk current image and the further shrunk reference image to generate a plurality of third motion vectors and a third reliability corresponding to the third motion vectors; where the operation of performing the first motion estimation further includes generating the first motion vectors and the first reliability based on the third reliability.

[0077] In summary, the image processing method 200 provided by the present application can provide the motion estimation of a larger-size image based on the motion estimation result of a smaller-size image. In addition to providing motion vectors, the reliability of the motion vectors is also provided as a reference, so that the image processing method 200 can adjust the intervention degree (e.g., weight) of the motion vectors in the next motion estimation based on the reliability of the motion vectors. In addition, the image processing method 200 can also use the reliability of the motion vectors as a reference for generating a compensated frame image in the motion compensation operation. Similarly, the image processing method 200 can adjust the intervention degree (e.g., weight) of the motion vectors in the motion compensation based on the reliability of the motion vectors. Through the motion estimation and motion compensation operations of multiple-size images, the efficiency and accuracy of obtaining motion vectors can be improved. Therefore, under the condition of maintaining or reducing the computing cost, the situation of image fragmentation, fringing or jerks can be effectively reduced.

[0078] Although several embodiments are described in detail above as examples, the image processing apparatus and method proposed by the present application can also be implemented by other systems, hardware, software, storage media or their combinations. Therefore, the protection scope of the present application should not be limited to the specific implementation manners described in the embodiments of the present application, but should be determined by the scope defined in the appended claims.

[0079] It is obvious to those of ordinary skill in the technical field to which the present application belongs that various modifications and changes can be made to the structure of the present application without departing from the scope or spirit of the present application. In view of the foregoing, the protection scope of the present application also covers the modifications and changes made within the scope of the appended claims.

Symbol Description

Claims

1. An image processing apparatus, characterized in that, comprising: a memory for storing a current image and a reference image; and a processor coupled to the memory for performing the following operations: scaling down the current image and the reference image to respectively generate a scaled-down current image and a scaled-down reference image; performing a first motion estimation on the scaled-down current image and the scaled-down reference image to generate a plurality of first motion vectors and a first reliability corresponding to the first motion vectors; performing an nth motion estimation on the current image and the reference image based on the first motion vectors and the first reliability to generate a plurality of nth motion vectors and an nth reliability corresponding to the nth motion vectors; and generating an interpolated frame image between the current image and the reference image based on the nth motion vectors and the nth reliability.

2. The image processing apparatus according to claim 1, characterized in that, performing the first motion estimation includes: calculating the first reliability based on the first motion vectors and a plurality of sums of absolute differences corresponding to the first motion vectors, wherein the sum of absolute differences is positively correlated with the first reliability.

3. The image processing apparatus according to claim 2, wherein calculating the first reliability based on the first motion vectors and the sum of absolute differences includes: calculating the first reliability based on a regional motion vector, a global motion vector, the first motion vectors and the sum of absolute differences, wherein a first correlation degree between the first motion vector and the regional motion vector is positively correlated with the first reliability, and a second correlation degree between the first motion vector and the global motion vector is positively correlated with the first reliability.

4. The image processing apparatus according to claim 1, wherein performing the nth motion estimation includes: generating an intermediate reliability based on the nth motion vectors; and generating the nth reliability based on the first reliability and the intermediate reliability, wherein the first reliability is positively correlated with the nth reliability, and the intermediate reliability is positively correlated with the nth reliability.

5. The image processing apparatus according to claim 1, wherein performing the nth motion estimation includes: calculating a plurality of matching degrees of a plurality of candidate vectors; and using one of the candidate vectors corresponding to the highest one of the matching degrees as one of the nth motion vectors.

6. The image processing apparatus according to claim 5, wherein calculating the matching degrees includes: calculating a vector difference between one of the candidate vectors and one of the first motion vectors corresponding to the candidate vector; calculating a penalty value based on the vector difference and the first reliability, wherein the penalty value is positively correlated with the vector difference and the first reliability; and reducing one of the matching degrees of the candidate vector based on the penalty value.

7. The image processing apparatus according to claim 1, wherein the nth motion estimation includes a plurality of scanning operations, and the scanning operations include: calculating a search window size based on the first reliability, wherein there is a linear correlation between the first reliability and the search window size; and Generate a plurality of candidate vectors within a search window having the search window size.

8. The image processing apparatus according to claim 1, wherein generating the interpolated frame image comprises: Selecting one of the current image and the reference image as a compensation target; Performing motion compensation based on the compensation target, the n-th reliability, and the n-th motion vector to generate the interpolated frame image corresponding to the compensation target, wherein the n-th reliability is negatively correlated with a similarity between the interpolated frame image and the compensated image.

9. The image processing apparatus according to claim 8, wherein there is an initial time distance between the interpolated frame image and the compensation target, and the processor is further configured to: Calculate a correction value based on the initial time distance and the n-th reliability, wherein the n-th reliability and the correction value are negatively correlated; and Adjust the initial time distance based on the correction value to determine a corrected time distance between the interpolated frame image and the compensation target.

10. An image processing method applicable to an electronic device, comprising: Reducing a current image and a reference image to respectively generate a reduced current image and a reduced reference image; Performing a first motion estimation on the reduced current image and the reduced reference image to generate a plurality of first motion vectors and a first reliability corresponding to the first motion vectors; Performing an n-th motion estimation on the current image and the reference image based on the first motion vectors and the first reliability to generate a plurality of n-th motion vectors and an n-th reliability corresponding to the n-th motion vectors; And Generating an interpolated frame image between the current image and the reference image based on the n-th motion vectors and the n-th reliability.

Images