Image Processing Method, Apparatus, and Storage Medium

Abstract

The present disclosure relates to an image processing method, apparatus, and storage medium. The method mainly includes: determining motion information of pixels of a first image frame based on an initial optical flow of the pixels of the first image frame in a sequence of images to be processed; projecting to obtain a second image frame based on the motion information of the pixels of the first image frame and the first image frame; determining a calibration parameter according to pixel values of pixels of the second image frame and pixel values of pixels of a third image frame adjacent to the first image frame, and calibrating each initial optical flow based on the calibration parameter. Since the second image frame is a subsequent image frame projected based on the initial optical flow and the first image frame, and the third image frame is the original subsequent image frame, based on the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame, the deviation degree of the initial optical flow can be determined, and then the initial optical flow can be calibrated to improve the accuracy of the optical flow.

Description

Technical Field

[0001] The present disclosure relates to the field of image technology, and in particular, to an image processing method, apparatus, and storage medium. Background Art

[0002] Optical flow plays a very important role in computer vision. Analyzing moving objects in an image sequence can help obtain the temporal characteristics, position information, structural information, etc. of the objects. Usually, directly calculating the motion field of an object is complex and difficult. Each pixel point in an image sequence implicitly contains rich motion information, and optical flow precisely represents the motion information of each pixel point. Therefore, optical flow is an essential and important means for motion analysis in an image sequence.

[0003] Optical flow has a very wide range of applications in computer vision. For example, in action recognition, optical flow is used to extract temporal domain features, thereby enabling action classification; in autonomous driving and simultaneous localization and mapping (SLAM), optical flow is used to track and re - locate objects; in video compression and video super - frame rate, optical flow is used for motion estimation, and then motion compensation is performed based on the motion estimation.

[0004] It can be seen that optical flow plays a crucial role in the above - mentioned examples. If accurate optical flow is not estimated, it will directly affect the quality of the final result. Therefore, it is very necessary to obtain more accurate optical flow and further process the accuracy of optical flow. Summary of the Invention

[0005] The present disclosure provides an image processing method, apparatus, and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including:

[0007] Determine the motion information of pixels of the first image frame based on the initial optical flow of the pixels of the first image frame in the image sequence to be processed;

[0008] Project to obtain a second image frame based on the motion information of the pixels of the first image frame and the first image frame;

[0009] Determine a calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of a third image frame adjacent to the first image frame in the image sequence;

[0010] Calibrate the initial optical flow based on the calibration parameter.

[0011] Optionally, determining a calibration parameter based on pixel values of pixels of the second image frame and pixel values of pixels of a third image frame adjacent to the first image frame in the image sequence to be processed includes:

[0012] Determining a calibration parameter corresponding to the initial optical flow according to a difference between pixel values of pixels of the second image frame and pixel values of pixels of the third image frame;

[0013] wherein the calibration parameter is used to characterize a deviation degree of the initial optical flow.

[0014] Optionally, calibrating the initial optical flow based on the calibration parameter includes:

[0015] When the calibration parameter is greater than or equal to a set threshold, weighting the initial optical flow based on the calibration parameter to obtain a calibrated optical flow.

[0016] Optionally, calibrating the initial optical flow based on the calibration parameter includes:

[0017] When the calibration parameter is less than the set threshold, determining a current pixel corresponding to the initial optical flow and pixels adjacent to the current pixel;

[0018] Calibrating the initial optical flow of the current pixel according to calibrated optical flows of pixels adjacent to the current pixel.

[0019] Optionally, calibrating the initial optical flow of the current pixel according to calibrated optical flows of pixels adjacent to the current pixel includes:

[0020] Weighting the calibrated optical flows of pixels adjacent to the current pixel based on a set weight to obtain a weighted value;

[0021] Obtaining a calibrated optical flow of the current pixel according to a ratio between a sum of the weighted values and a number of pixels adjacent to the current pixel.

[0022] Optionally, the method further includes:

[0023] Processing the calibrated optical flow by using variational method based on an image gradient of the first image frame to obtain a target optical flow.

[0024] According to a second aspect of embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0025] A first obtaining module configured to determine motion information of pixels of the first image frame based on an initial optical flow of pixels of the first image frame in an image sequence to be processed;

[0026] A projection module, configured to project a second image frame based on the motion information of the pixels of the first image frame and the first image frame;

[0027] A determination module, configured to determine a calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of a third image frame adjacent to the first image frame in the image sequence;

[0028] A calibration module, configured to calibrate the initial optical flow based on the calibration parameter.

[0029] Optionally, the determination module is further configured to:

[0030] Determine the calibration parameter corresponding to the initial optical flow according to the difference between the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame;

[0031] Wherein, the calibration parameter is used to characterize the deviation degree of the initial optical flow.

[0032] Optionally, the calibration module is further configured to:

[0033] When the calibration parameter is greater than or equal to a set threshold, weight the initial optical flow based on the calibration parameter to obtain a calibrated optical flow.

[0034] Optionally, the calibration module is further configured to:

[0035] When the calibration parameter is less than the set threshold, determine the current pixel corresponding to the initial optical flow and the pixels adjacent to the current pixel;

[0036] Calibrate the initial optical flow of the current pixel according to the calibrated optical flow of the pixels adjacent to the current pixel.

[0037] Optionally, the calibration module is further configured to:

[0038] Weight the calibrated optical flow of the pixels adjacent to the current pixel based on a set weight to obtain a weighted value;

[0039] Obtain the calibrated optical flow of the current pixel according to the ratio between the sum of the weighted values and the number of the pixels adjacent to the current pixel.

[0040] Optionally, the apparatus further includes:

[0041] A processing module, configured to process the calibrated optical flow by using variational method based on the image gradient of the first image frame to obtain a target optical flow.

[0042] According to a third aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0043] Processor;

[0044] A memory configured to store processor-executable instructions;

[0045] Wherein, the processor is configured to: when executed, implement the steps in any one of the above-mentioned first aspect of the image processing method.

[0046] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an image processing device, enabling the device to execute any one of the above-mentioned first aspect of the image processing method.

[0047] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0048] As can be seen from the above embodiments, after obtaining the initial optical flow, the present disclosure can project the initial optical flow and the first image frame to obtain the second image frame, and determine the calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame adjacent to the first image frame, and calibrate the initial optical flow based on the calibration parameter. Since the second image frame is the subsequent frame image obtained by projecting the initial optical flow and the first image frame, and the third image frame is the original subsequent frame image, based on the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame, the deviation degree of the initial optical flow can be determined, and then the initial optical flow can be calibrated to improve the accuracy of the optical flow.

[0049] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0051] Figure 1 It is a schematic flowchart of an image processing method shown according to an exemplary embodiment.

[0052] Figure 2 It is a schematic diagram of an image frame shown according to an exemplary embodiment Figure 1 .

[0053] Figure 3 It is a schematic diagram of local optical flow magnification shown according to an exemplary embodiment.

[0054] Figure 4 It is a schematic diagram of filling shown according to an exemplary embodiment.

[0055] Figure 5 It is a schematic diagram of an image frame shown according to an exemplary embodiment Figure 2 .

[0056] Figure 6 It is a schematic diagram of an image frame shown according to an exemplary embodiment Figure 3 .

[0057] Figure 7 It is a schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 1 .

[0058] Figure 8 It is a schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 2 .

[0059] Figure 9 It is a schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 3 .

[0060] Figure 10 It is a schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 4 .

[0061] Figure 11 It is a block diagram of an image processing apparatus shown according to an exemplary embodiment.

[0062] Figure 12 It is a block diagram of an image processing apparatus 1200 shown according to an exemplary embodiment.

[0063] Figure 13 It is another block diagram of an image processing apparatus 1300 shown according to an exemplary embodiment. Detailed implementation manners

[0064] Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] An image processing method is provided in an embodiment of the present disclosure Figure 1 It is a schematic flowchart of an image processing method shown according to an exemplary embodiment, as Figure 1 shown, and the method mainly includes the following steps

[0066] In step 101, based on the initial optical flow of the pixels of the first image frame in the image sequence to be processed, determine the motion information of the pixels of the first image frame

[0067] In step 102, based on the motion information of the pixels of the first image frame and the first image frame, a second image frame is projected.

[0068] In step 103, calibration parameters are determined according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame adjacent to the first image frame in the image sequence.

[0069] In step 104, the initial optical flow is calibrated based on the calibration parameters.

[0070] The image processing method involved in the embodiments of the present disclosure can be applied to electronic devices, where the electronic devices include mobile terminals and fixed terminals. Among them, the mobile terminals include: mobile phones, tablet computers, laptop computers, etc.; the fixed terminals include: personal computers. In other alternative embodiments, the image processing method can also run on network-side devices, where the network-side devices include: servers, processing centers, etc.

[0071] In the embodiments of the present disclosure, an initial optical flow can be obtained through a preset optical flow estimation algorithm. For example, an initial optical flow can be obtained based on a convolutional neural network, or an initial optical flow can be obtained using variational methods. Here, the initial optical flow can be a two-layer matrix, where one layer represents u as the horizontal motion vector and the other layer represents v as the vertical motion vector. Figure 2 It is a schematic diagram of an image frame shown according to an exemplary embodiment Figure 1 , Figure 2 In (a), it represents the first image frame, in (b), it represents the third image frame, and in (c), it represents the initial optical flow of the first image frame. Figure 3 It is a schematic diagram of local optical flow amplification shown according to an exemplary embodiment, Figure 3 In (a), it represents the horizontal motion vector, and in (b), it represents the vertical motion vector.

[0072] After obtaining the initial optical flow, the motion information of the pixels of the first image frame can be determined based on the initial optical flow, and a second image frame can be projected based on the motion information of the pixels of the first image frame and the first image frame. Among them, the motion information includes: horizontal motion vector and vertical motion vector. The calculation formula for obtaining the second image frame is as follows:

[0073]

[0074] In formula (1), It represents the second image frame, that is, the subsequent frame image of the projection. I1 represents the first image frame, and f(x) represents the initial optical flow, where f(x) ∈ {u, v}, u represents the horizontal motion vector, v represents the vertical motion vector, and near() represents taking the nearest integer coordinates.

[0075] After obtaining the second image frame, the calibration parameter can be determined based on the pixel values of the pixels in the second image frame and the pixel values of the pixels in the third image frame adjacent to the first image frame in the image sequence to be processed, and the initial optical flow can be calibrated based on the calibration parameter. In some embodiments, the calibration parameter can be directly obtained based on the difference between the pixel values of each pixel in the second image frame and the pixel values of each pixel in the third image frame. Of course, the calibration parameter can also be determined based on other methods. For example, each pixel value in the second image frame and the third image frame is weighted, and then the calibration parameter is determined based on the difference between the weighted pixel values. Specific limitations are not made here.

[0076] In some embodiments, the image sequence to be processed can be a video to be processed composed of multiple consecutive image frames, or an image sequence composed of image frames arranged in a set order. Specific limitations are not made here.

[0077] In the embodiments of the present disclosure, after obtaining the initial optical flow, the second image frame can be projected based on the initial optical flow and the first image frame, and the calibration parameter can be determined according to the pixel values of the pixels in the second image frame and the pixel values of the pixels in the third image frame adjacent to the first image frame, and the initial optical flow can be calibrated based on the calibration parameter. Since the second image frame is the subsequent frame image projected based on the initial optical flow and the first image frame, and the third image frame is the original subsequent frame image, based on the pixel values of the pixels in the second image frame and the pixel values of the pixels in the third image frame, the deviation degree of the initial optical flow can be determined, and then the initial optical flow can be calibrated to improve the accuracy of the optical flow.

[0078] In some embodiments, the determining the calibration parameter according to the pixel values of the pixels in the second image frame and the pixel values of the pixels in the third image frame adjacent to the first image frame in the image sequence includes:

[0079] Determining the calibration parameter corresponding to the initial optical flow according to the difference between the pixel values of the pixels in the second image frame and the pixel values of the pixels in the third image frame;

[0080] Wherein, the calibration parameter is used to characterize the deviation degree of the initial optical flow.

[0081] In some embodiments, the difference between the second image frame and the third image frame is negatively correlated with the calibration parameter corresponding to the initial optical flow, and the deviation degree of the initial optical flow is negatively correlated with the calibration parameter. That is to say, the smaller the difference between the second image frame and the third image frame, the higher the calibration parameter corresponding to the initial optical flow, and the smaller the deviation degree of the initial optical flow; the larger the difference between the second image frame and the third image frame, the lower the calibration parameter corresponding to the initial optical flow, and the larger the deviation degree of the initial optical flow. Among them, the calibration parameter can be represented by confidence, and no specific limitation is made here.

[0082] In the embodiments of the present disclosure, the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame can be first determined, and then the calibration parameters corresponding to the respective initial optical flows are determined based on the differences between the respective pixels. In some embodiments, the calculation formula of the calibration parameter is as follows:

[0083]

[0084] where t represents the calibration parameter, max() represents taking the maximum value, e is the natural exponent, represents the pixel value of the pixel of the second image frame, I2 represents the pixel value of the pixel of the third image frame, when I2 is close to the value of , the calibration parameter is higher, and when I2 differs greatly from the value of , the calibration parameter is lower.

[0085] In the embodiments of the present disclosure, since the calibration parameter is determined based on the difference between the pixel values of the pixels of the second image frame and the third image frame, the deviation degree of the initial optical flow is characterized by the calibration parameter, and the initial optical flow is calibrated based on the calibration parameter, which can improve the accuracy of the calibrated optical flow.

[0086] In some embodiments, the calibrating the initial optical flow based on the calibration parameter includes:

[0087] When the calibration parameter is greater than or equal to a set threshold, the initial optical flow is weighted based on the calibration parameter to obtain a calibrated optical flow.

[0088] Since the initial optical flows and calibration parameters corresponding to the respective pixels are different, in the embodiments of the present disclosure, after obtaining the calibration parameter corresponding to the initial optical flow, the initial optical flow can be directly calibrated based on the calibration parameter, or the calibration parameter can be weighted, and the initial optical flow can be calibrated based on the weighted calibration parameter, etc., and no specific limitation is made here.

[0089] In some embodiments, the calibration parameter can be compared with a set threshold, and when the calibration parameter is greater than or equal to the set threshold, the initial optical flow can be directly calibrated based on this calibration parameter. In other embodiments, when the calibration parameter is less than the set threshold, the calibration parameter of the initial optical flow is set to zero. The formula for determining the calibration parameter is as follows:

[0090]

[0091] In formula (3), c(x) represents the finally determined calibration parameter, t represents the current calibration parameter, and the set threshold is 1e -3 . In other alternative embodiments, the set threshold can also be other values, which can be determined according to empirical values or through a set algorithm, and are not specifically limited herein.

[0092] In the embodiments of the present disclosure, since the calibration parameter is negatively correlated with the difference between the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame, when the calibration parameter is greater than or equal to the set threshold, it indicates that the difference between the second image frame and the third image frame is small. At this time, the initial optical flow can be directly calibrated based on the calibration parameter, thereby improving the accuracy of the calibrated optical flow.

[0093] In the embodiments of the present disclosure, after obtaining the calibration parameter corresponding to the initial optical flow, the initial optical flow can be weighted based on the calibration parameter to obtain a calibrated optical flow. For example, a dot product calculation can be performed on the initial optical flow and this calibration parameter to obtain a calibrated optical flow. The formula for the calibrated optical flow is as follows:

[0094] f c (x) = f(x).c(x) (4)

[0095] In formula (4), f c represents the calibrated optical flow, f represents the initial optical flow, and c represents the calibration parameter.

[0096] In the embodiments of the present disclosure, since the calibration parameter is determined based on the difference between the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame, the deviation degree of the initial optical flow is characterized by the calibration parameter, and the calibrated optical flow is obtained based on the dot product calculation between the calibration parameter and the initial optical flow, which can improve the accuracy of the calibrated optical flow.

[0097] In some embodiments, calibrating the corresponding initial optical flow according to the calibration parameter includes:

[0098] When the calibration parameter is less than the set threshold, determining the current pixel corresponding to the initial optical flow and the pixels adjacent to the current pixel;

[0099] Calibrate the initial optical flow of the current pixel according to the calibrated optical flow of the pixels adjacent to the current pixel.

[0100] When the calibration parameter is less than the set threshold, it indicates that the difference between the second image frame and the third image frame is large. The reason for the large difference between the second image frame and the third image frame may be abnormal initial optical flow. If the initial optical flow is directly calibrated based on the calibration parameter at this time, the calibrated optical flow may be inaccurate.

[0101] Since when the calibration parameter is less than the set threshold, it indicates that the initial optical flow corresponding to this point may be incorrect. In the embodiments of the present disclosure, when the calibration parameter is less than the set threshold, the calibration parameter will be set to zero. According to formula (4), when the calibration parameter is zero, the obtained calibrated optical flow is also zero, and the pixel point corresponding to this calibrated optical flow is a hole. In the embodiments of the present disclosure, a soft fusion method can be used to fill the position of the hole.

[0102] For example, the current pixel corresponding to the initial optical flow and the pixels adjacent to the current pixel can be determined; the initial optical flow of the current pixel is calibrated according to the calibrated optical flow of the pixels adjacent to the current pixel. Figure 4 is a filling schematic diagram shown according to an exemplary embodiment, as Figure 4 shown, the position where the current pixel 301 is located is a hole. There are four pixels adjacent to the current pixel 301, namely the first pixel 302, the second pixel 303, the third pixel 304, and the fourth pixel 305. In the implementation process, the initial optical flow of the current pixel 301 can be calibrated based on the first calibrated optical flow of the first pixel 302, the second calibrated optical flow of the second pixel 303, the third calibrated optical flow of the third pixel 304, and the fourth calibrated optical flow of the fourth pixel 305.

[0103] For example, the calibrated optical flow of the current pixel can be obtained based on the average value of the first calibrated optical flow, the second calibrated optical flow, the third calibrated optical flow, and the fourth calibrated optical flow. For another example, the first calibrated optical flow, the second calibrated optical flow, the third calibrated optical flow, and the fourth calibrated optical flow are sorted, and the calibrated optical flow of the current pixel is determined based on the sorting result, etc., which are not specifically limited herein.

[0104] In some embodiments, the calibrating the initial optical flow of the current pixel according to the calibrated optical flow of the pixels adjacent to the current pixel includes:

[0105] Weight the calibrated optical flow of the pixels adjacent to the current pixel based on a set weight to obtain a weighted value;

[0106] The calibrated optical flow of the current pixel is obtained according to the ratio between the sum of the respective weighted values and the number of pixels adjacent to the current pixel.

[0107] Here, the calculation formula for the calibrated optical flow of the current pixel is as follows:

[0108]

[0109] In formula (5), z(x) represents the calibrated optical flow of the current pixel, v right represents the first calibrated optical flow of the first pixel, v down represents the second calibrated optical flow of the second pixel, v left represents the third calibrated optical flow of the third pixel, v up represents the fourth calibrated optical flow of the fourth pixel, and divisor represents the number of pixels adjacent to the current pixel.

[0110] In the embodiments of the present disclosure, the way of directly using the nearest neighbor filling method can be adopted to determine the calibrated optical flow of the current pixel, which can make the finally determined optical flow more natural.

[0111] In some embodiments, the method further includes:

[0112] Based on the image gradient of the first image frame, the calibrated optical flow is processed by using the variational method to obtain the target optical flow.

[0113] Here, the optical flow estimation method based on the variational method assumes that the brightness of the image remains consistent over continuous time. The calculation formula of the variational method is as follows:

[0114] I(x,y,t) = I(x + u, y + v, t + 1) (6);

[0115] After performing Taylor expansion and simplification on formula (6), the following formula is obtained:

[0116] I x u + I y v + I t = 0(2) (7);

[0117] In formula (7), Ix represents the horizontal gradient of the image, Iy represents the vertical gradient of the image, It represents the temporal gradient of the image, u represents the horizontal optical flow, v represents the vertical optical flow, and is represented by a vector as:

[0118]

[0119] Among them, ω = [u, v] T .

[0120] When the optical flow is in the normal direction, since the direction cannot be estimated, the optical flow is uncertain at this time, that is, the "aperture problem" of the optical flow. To solve this problem, a smoothness constraint term is introduced, and it is assumed that the optical flow field is piecewise smooth. In the related art, the variational energy equation can be used to calculate the optical flow u and v. The energy equation includes two terms, namely: the data term and the smoothness constraint term. The formula is as follows:

[0121]

[0122] In formula (9), the first term is the data term, and the second term is the smoothness constraint term.

[0123] In the embodiments of the present disclosure, a gradient term can be constructed based on the image gradient of the first image frame and the calibrated optical flow, and an energy equation in variational method can be constructed based on the data term, the gradient term, and the smoothness constraint term. For example, a gradient term can be constructed based on the first product between the horizontal gradient of the first image frame and the calibrated horizontal optical flow, the second product between the vertical gradient of the first image frame and the calibrated vertical optical flow, and the temporal gradient of the first image frame. For another example, the first product, the second product, and the temporal gradient can be added to obtain the gradient term.

[0124] In the embodiments of the present disclosure, a gradient term is added to the energy equation in the related art. The formula is as follows:

[0125]

[0126] In formula (10), Ix represents the horizontal gradient of the image, Iy represents the vertical gradient of the image, It represents the temporal gradient of the image. The first term is the data term, the second term is the gradient term, and the third term is the smoothness constraint term, where represents the gradient.

[0127] In the embodiments of the present disclosure, after obtaining the calibrated optical flow, the calibrated optical flow can be processed by using variational method based on the image gradient of the first image frame to obtain the target optical flow, and the target optical flow obtained after variational is more refined.

[0128] In the implementation process, two sets of image frames can be selected for testing. Let the two sets of image frames be S Ⅰ group and S Ⅱ group. Figure 5 is a schematic diagram of an image frame shown according to an exemplary embodiment Figure 2 , Figure 5 in which part (a) represents the first image frame of S Ⅰ group, part (b) represents the third image frame of S Ⅰ group, and part (c) represents the initial optical flow of the first image frame of S Ⅰ group. Figure 6Schematic diagram of an image frame shown according to an exemplary embodiment Figure 3 , Figure 6 Part (a) in Ⅱ represents the first image frame of the S Ⅱ group, part (b) represents the third image frame of the S Ⅱ group, and part (c) represents the initial optical flow of the first image frame of the S

[0129] Figure 7 Schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 1 , as Figure 7 shown, the initial optical flow shown in Figure 5 can be calibrated based on the SimpleFlow algorithm, Figure 7 where part (a) in

[0130] Figure 8 represents the initial optical flow of the first image frame, and part (b) represents the calibrated optical flow obtained after calibrating the initial optical flow.

[0130] Figure 8 Schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 2 , as Figure 8 shown, the initial optical flow shown in Figure 5 can be calibrated based on the SpyNet algorithm, Figure 8 where part (a) in

[0131] Figure 9 represents the initial optical flow of the first image frame, and part (b) represents the calibrated optical flow obtained after calibrating the initial optical flow.

[0131] Figure 9 Schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 3 , as Figure 9 shown, the initial optical flow shown in Figure 6 can be calibrated based on the SimpleFlow algorithm, Figure 9 where part (a) in represents the initial optical flow of the first image frame, and part (b) represents the calibrated optical flow obtained after calibrating the initial optical flow.

[0132] Figure 10 Schematic diagram of the result of calibrating the initial optical flow based on the prior art Figure 4 , as Figure 10 shown, the initial optical flow shown in Figure 6 can be calibrated based on the SpyNet algorithm, Figure 10 where part (a) in

[0133] During the implementation process, the point error, angular error, and standard deviation of the initial optical flow, calibrated optical flow, and true optical flow can be calculated separately. Four error threshold values are set, which are 0.5, 1.0, 2.0, and 5.0 respectively. The results are shown in the following table:

[0134] Table 1 S Ⅰ Comparison Table of Optical Flow Calibration Results for Group S

[0135]

[0136]

[0137] Table 2 S Ⅱ Comparison Table of Optical Flow Calibration Results for Group S

[0138]

[0139] It can be seen from the four groups of comparison data that whether the variational method is used to estimate the optical flow or the convolutional neural network is used to estimate the optical flow (initial optical flow), there are errors. After the optical flow (initial optical flow) estimated by the two methods is corrected and refined, it can be seen that the point error (endpointError) and angular error (angularError) of the calibrated optical flow are significantly reduced. Especially when the threshold value is 0.5, the error reduction is the most obvious. Calculate the standard deviation of the difference between the estimated optical flow (initial optical flow) and the true optical flow, and the standard deviation of the difference between the corrected and refined optical flow (calibrated optical flow) and the true optical flow. By comparison, it can be seen that the standard deviation of the optical flow after correction and refinement is smaller, indicating that the optical flow after correction and refinement is closer to the true optical flow, and the optical flow accuracy has been improved.

[0140] Figure 11 is a block diagram of an image processing device shown according to an exemplary embodiment. As Figure 11 shown, the device 1100 mainly includes:

[0141] The first acquisition module 1101 is configured to determine the motion information of the pixels of the first image frame based on the initial optical flow of the pixels of the first image frame in the image sequence to be processed;

[0142] The projection module 1102 is configured to project the first image frame based on the motion information of the pixels of the first image frame and the first image frame to obtain a second image frame;

[0143] The determination module 1103 is configured to determine the calibration parameter according to the pixel value of the pixels of the second image frame and the pixel value of the pixels of the third image frame adjacent to the first image frame in the image sequence;

[0144] The calibration module 1104 is configured to calibrate the initial optical flow based on the calibration parameter.

[0145] In some embodiments, the determining module 1103 is further configured to:

[0146] Determine a calibration parameter corresponding to the initial optical flow according to a difference between pixel values of pixels of the second image frame and pixel values of pixels of the third image frame;

[0147] Wherein, the calibration parameter is used to characterize a deviation degree of the initial optical flow.

[0148] In some embodiments, the calibration module 1104 is further configured to:

[0149] When the calibration parameter is greater than or equal to a set threshold, weight the initial optical flow based on the calibration parameter to obtain a calibrated optical flow.

[0150] In some embodiments, the calibration module 1104 is further configured to:

[0151] When the calibration parameter is less than the set threshold, determine a current pixel corresponding to the initial optical flow and pixels adjacent to the current pixel;

[0152] Calibrate the initial optical flow of the current pixel according to the calibrated optical flow of pixels adjacent to the current pixel.

[0153] In some embodiments, the calibration module 1104 is further configured to:

[0154] Weight the calibrated optical flow of pixels adjacent to the current pixel based on a set weight to obtain a weighted value;

[0155] Obtain the calibrated optical flow of the current pixel according to a ratio between a sum of the weighted values and a number of pixels adjacent to the current pixel.

[0156] In some embodiments, the apparatus 1100 further includes:

[0157] A processing module, configured to process the calibrated optical flow by using a variational method based on an image gradient of the first image frame to obtain a target optical flow.

[0158] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0159] Figure 12FIG. 0 is a block diagram of an image processing apparatus 1200 shown in accordance with an exemplary embodiment. For example, apparatus 1200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0160] Referring Figure 12 to FIG. 5, apparatus 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power component 1206, a multimedia component 1208, an audio component 1210, an input / output (I / O) interface 1212, a sensor component 1214, and a communication component 1216.

[0161] The processing component 1202 generally controls the overall operation of apparatus 1200, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above-described methods. In addition, the processing component 1202 may include one or more modules to facilitate interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate interaction between the multimedia component 1208 and the processing component 1202.

[0162] The memory 1204 is configured to store various types of data to support the operation of device 1200. Examples of such data include instructions for any application or method operating on apparatus 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0163] The power component 1206 provides power to the various components of apparatus 1200. The power component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for apparatus 1200.

[0164] The multimedia component 1208 includes a screen that provides an output interface between the device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of a touch or swipe action but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the device 1200 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0165] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC) that is configured to receive external audio signals when the device 1200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1204 or transmitted via the communication component 1216. In some embodiments, the audio component 1210 further includes a speaker for outputting audio signals.

[0166] The I / O interface 1212 provides an interface between the processing component 1202 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0167] The sensor component 1214 includes one or more sensors for providing an assessment of the status of various aspects of the device 1200. For example, the sensor component 1214 can detect the on / off state of the device 1200, the relative positioning of components, such as the display and keypad of the device 1200, the sensor component 1214 can also detect a change in the position of the device 1200 or a component of the device 1200, the presence or absence of user contact with the device 1200, the orientation or acceleration / deceleration of the device 1200, and the temperature change of the device 1200. The sensor component 1214 can include a proximity sensor that is configured to detect the presence of nearby objects without any physical contact. The sensor component 1214 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1214 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0168] The communication component 1216 is configured to facilitate communication between the device 1200 and other devices in a wired or wireless manner. The device 1200 can access a communication standard-based wireless network, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0169] In an exemplary embodiment, the device 1200 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0170] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, and the above instructions can be executed by a processor 1220 of the device 1200 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0171] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an image processing device, enables the image processing device to execute an image processing method, the method including:

[0172] Determining motion information of pixels of a first image frame based on an initial optical flow of the pixels of the first image frame in a sequence of images to be processed;

[0173] Projecting to obtain a second image frame based on the motion information of the pixels of the first image frame and the first image frame;

[0174] Determining a calibration parameter according to pixel values of pixels of the second image frame and pixel values of pixels of a third image frame adjacent to the first image frame in the image sequence;

[0175] Calibrating each of the initial optical flows based on the calibration parameter.

[0176] Figure 13Another block diagram for an image processing apparatus 1300 is shown according to an exemplary embodiment. For example, the apparatus 1300 may be provided as a server. Referring to Figure 13 , the apparatus 1300 includes a processing component 1322, which further includes one or more processors, and memory resources represented by a memory 1332 for storing instructions executable by the processing component 1322, such as application programs. The application programs stored in the memory 1332 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1322 is configured to execute instructions to perform the above-described image processing method, the method including:

[0177] Determining motion information of pixels of the first image frame based on an initial optical flow of pixels of the first image frame in a sequence of images to be processed;

[0178] Projecting to obtain a second image frame based on the motion information of pixels of the first image frame and the first image frame;

[0179] Determining a calibration parameter according to pixel values of pixels of the second image frame and pixel values of pixels of a third image frame adjacent to the first image frame in the sequence of images to be processed;

[0180] Calibrating the initial optical flow based on the calibration parameter.

[0181] The apparatus 1300 may further include a power component 1326 configured to perform power management of the apparatus 1300, a wired or wireless network interface 1350 configured to connect the apparatus 1300 to a network, and an input / output (I / O) interface 1358. The apparatus 1300 may operate based on an operating system stored in the memory 1332, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0182] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed in this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present invention are pointed out by the following claims.

[0183] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Determine the motion information of the pixels of the first image frame based on the initial optical flow of the pixels of the first image frame in the image sequence to be processed; Project to obtain a second image frame based on the motion information of the pixels of the first image frame and the first image frame; Determine a calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame adjacent to the first image frame in the image sequence; When the calibration parameter is greater than or equal to a set threshold, weight the initial optical flow based on the calibration parameter to obtain a calibrated optical flow; When the calibration parameter is less than the set threshold, determine the current pixel corresponding to the initial optical flow and the pixels adjacent to the current pixel; Weight the calibrated optical flow of the pixels adjacent to the current pixel based on a set weight to obtain a weighted value; obtain the calibrated optical flow of the current pixel according to the ratio between the sum of the weighted values and the number of pixels adjacent to the current pixel.

2. The method according to claim 1, wherein The determining the calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame adjacent to the first image frame in the image sequence includes: Determine the calibration parameter corresponding to the initial optical flow according to the difference between the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame; Wherein, the calibration parameter is used to characterize the deviation degree of the initial optical flow.

3. The method according to claim 1 or 2, characterized in that The method further includes: Process the calibrated optical flow using the variational method based on the image gradient of the first image frame to obtain a target optical flow.

4. An image processing apparatus, characterized in that, Including: A first acquisition module configured to determine the motion information of the pixels of the first image frame based on the initial optical flow of the pixels of the first image frame in the image sequence to be processed; A projection module configured to project to obtain a second image frame based on the motion information of the pixels of the first image frame and the first image frame; A determination module configured to determine a calibration parameter according to the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame adjacent to the first image frame in the image sequence; A calibration module configured to, when the calibration parameter is greater than or equal to a set threshold, weight the initial optical flow based on the calibration parameter to obtain a calibrated optical flow; When the calibration parameter is less than the set threshold, determine the current pixel corresponding to the initial optical flow and the pixels adjacent to the current pixel; Weight the calibrated optical flow of the pixels adjacent to the current pixel based on a set weight to obtain a weighted value; obtain the calibrated optical flow of the current pixel according to the ratio between the sum of the weighted values and the number of pixels adjacent to the current pixel.

5. The device according to claim 4, characterized in that, The determination module is further configured to: Determine the calibration parameter corresponding to the initial optical flow according to the difference between the pixel values of the pixels of the second image frame and the pixel values of the pixels of the third image frame; Wherein, the calibration parameter is used to characterize the deviation degree of the initial optical flow.

6. The device according to claim 4 or 5, characterized in that, The device further includes: A processing module configured to process the calibrated optical flow using the variational method based on the image gradient of the first image frame to obtain a target optical flow.

7. An image processing apparatus, characterized in that, Including: A processor; A memory configured to store processor-executable instructions; Wherein, the processor is configured to: when executed, implement the steps in any one of the image processing methods in claims 1 to 3 above.

8. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an image processing device, enabling the device to execute any one of the image processing methods in claims 1 to 3 above.

Images