Image processing method, device, electronic device and storage medium

Abstract

The disclosed embodiment discloses an image processing method, which includes: after collecting image frames through an image acquisition device, based on the pixel values ​​of the collected nth image frame and the n+1th image frame, obtaining a first optical flow estimation value corresponding to the nth image frame; based on the nth image frame and the first optical flow estimation value, generating an nth prediction frame; determining a weight value according to the difference between the pixel value of the nth prediction frame and the pixel value of the n+1th frame image; correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value; based on the collected image frame and the second optical flow estimation value, generating a target prediction frame at a target time domain position; inserting the target prediction frame at the target time domain position. In the disclosed embodiment, compared with the method of directly generating a prediction frame using the uncorrected first optical flow estimation value without correcting the first optical flow estimation value using the weight value, the error of the first optical flow estimation value can be reduced, so that the generated prediction frame is more accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology but is not limited to the field of terminal technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Optical flow is a very important parameter used in computer vision technology, which directly affects the effect of image processing. Optical flow refers to the speed corresponding to the movement of patterns in time-varying images. Optical flow is widely used in various application scenarios in computer vision. For example, in behavior recognition scenarios, optical flow is used to extract time domain features, in unmanned driving scenarios, optical flow is used to track and locate targets, and in ultra-frame rate applications, optical flow is used for motion estimation. The accuracy of optical flow directly determines the effect of image processing in various application scenarios. Summary of the invention

[0003] The embodiments of the present disclosure disclose an image processing method, an apparatus, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0005] After the image frames are captured by the image capture device, a first optical flow estimation value corresponding to the nth image frame is obtained based on the pixel values ​​of the captured nth image frame and the n+1th image frame; wherein n is a natural number;

[0006] Generate an nth predicted frame based on the nth image frame and the first optical flow estimation value;

[0007] Determining a weight value according to a difference between a pixel value of the nth prediction frame and a pixel value of the n+1th image frame;

[0008] Correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value;

[0009] Generate a target prediction frame at the target time domain position based on the collected image frame and the second optical flow estimation value;

[0010] The target prediction frame is inserted at the target time domain position.

[0011] In one embodiment, determining the weight value according to the difference between the pixel value of the nth prediction frame and the pixel value of the (n+1)th image frame includes:

[0012] If the difference is greater than a set threshold, it is determined that the weight value is less than the weight threshold.

[0013] In one embodiment, the step of correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value includes:

[0014] Based on the product of the weight value and the first optical flow estimation value, a corrected second optical flow estimation value is obtained.

[0015] In one embodiment, the method further comprises:

[0016] If the corrected second optical flow estimation value is less than an estimation value threshold, the second optical flow estimation value less than the estimation value threshold is updated to a preset estimation value.

[0017] In one embodiment, the method further comprises:

[0018] Generate a second optical flow estimation value matrix according to second optical flow estimation values ​​of pixels in the nth image frame or the (n+1)th image frame;

[0019] If the corrected second optical flow estimation value is less than an estimation value threshold, updating the second optical flow estimation value less than the estimation value threshold to a preset estimation value includes:

[0020] If the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, the second optical flow estimation value of the Nth element is updated to: the average value between adjacent elements of the Nth element; N is a natural number.

[0021] In one embodiment, the method further comprises:

[0022] Obtain an optical flow estimation gradient value of the second optical flow estimation value;

[0023] Superimposing the optical flow estimation gradient value and the second optical flow estimation value to obtain a superimposed second optical flow estimation value;

[0024] The superimposed second optical flow estimation value is input into a global energy functional; wherein the global energy functional includes a data term and a smoothing term; and the global energy functional is solved to obtain an updated second optical flow estimation value.

[0025] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing device, characterized in that the device comprises a calculation module, a generation module, a determination module, a correction module and an insertion module; wherein:

[0026] The calculation module is used to: after the image frame is acquired by the image acquisition device, based on the pixel values ​​of the acquired nth image frame and the (n+1)th image frame, obtain a first optical flow estimation value corresponding to the nth image frame; wherein n is a natural number;

[0027] The generating module is used to: generate an nth predicted frame based on the nth image frame and the first optical flow estimation value;

[0028] The determination module is used to: determine the weight value according to the difference between the pixel value of the nth prediction frame and the pixel value of the (n+1)th image frame;

[0029] The correction module is used to: correct the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value;

[0030] The generating module is further used to: generate a target prediction frame at a target time domain position based on the collected image frame and the second optical flow estimation value;

[0031] The inserting module is used to insert the target prediction frame at the target time domain position.

[0032] In one embodiment, the determination module is further configured to configure: the difference is negatively correlated with the weight value determined according to the difference.

[0033] In one embodiment, the determining module is further used to:

[0034] If the difference is greater than a set threshold, it is determined that the weight value is less than the weight threshold.

[0035] In one embodiment, the correction module is further used to:

[0036] Based on the product of the weight value and the first optical flow estimation value, a corrected second optical flow estimation value is obtained.

[0037] In one embodiment, the device further includes an updating module, wherein the updating module is further configured to:

[0038] If the corrected second optical flow estimation value is less than an estimation value threshold, the second optical flow estimation value less than the estimation value threshold is updated to a preset estimation value.

[0039] In one embodiment,

[0040] The generating module is further used for:

[0041] Generate a second optical flow estimation value matrix according to second optical flow estimation values ​​of pixels in the nth image frame or the (n+1)th image frame;

[0042] The correction module is also used to: if the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, update the second optical flow estimation value of the Nth element to: the average value between adjacent elements of the Nth element; N is a natural number.

[0043] In one embodiment, the device further comprises an acquisition module, wherein:

[0044] The acquisition module is used to obtain the optical flow estimation gradient value of the second optical flow estimation value; superimpose the optical flow estimation gradient value with the second optical flow estimation value to obtain the superimposed second optical flow estimation value; input the superimposed second optical flow estimation value into the global energy function; wherein the global energy function includes a data term and a smoothing term; solve the global energy function to obtain the updated second optical flow estimation value.

[0045] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0046] processor;

[0047] a memory for storing instructions executable by the processor;

[0048] The processor is configured to implement the method described in any embodiment of the present disclosure when running the executable instructions.

[0049] According to a fourth aspect of an embodiment of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores a computer executable program, and when the executable program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0050] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0051] In an embodiment of the present disclosure, after an image frame is captured by an image capture device, a first optical flow estimation value corresponding to the nth image frame is obtained based on the pixel values ​​of the captured nth image frame and the n+1th image frame, and an nth prediction frame is generated based on the nth image frame and the first optical flow estimation value. Here, since the nth prediction frame is generated based on the nth image frame and the first optical flow estimation value, the n+1th image frame is the original image frame of the nth prediction frame. A weight value is determined based on the difference between the pixel values ​​of the nth prediction frame and the pixel values ​​of the n+1th image frame; the first optical flow estimation value is corrected based on the weight value to obtain a corrected second optical flow estimation value; a target prediction frame at a target time domain position is generated based on the captured image frame and the corrected second optical flow estimation value; and the target prediction frame is inserted at the target time domain position. In this way, the weight value can be determined according to the difference between the pixel value of the nth predicted frame and the pixel value of the original image frame, i.e., the n+1th image frame, and the first optical flow estimation value can be corrected using the weight value to obtain the corrected second optical flow estimation value. The corrected second optical flow estimation value can then be used to generate a predicted frame. Compared with the method of directly using the uncorrected first optical flow estimation value to generate a predicted frame without correcting the first optical flow estimation value using the weight value, the error of the first optical flow estimation value can be reduced, making the generated predicted frame more accurate.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0054] Figure 1 It is a flowchart of an image processing method according to an exemplary embodiment.

[0055] Figure 2 is a schematic diagram showing how to obtain optical flow according to an exemplary embodiment.

[0056] Figure 3 The figure is a schematic diagram of an optical flow vector according to an exemplary embodiment.

[0057] Figure 4 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0058] Figure 5 is a schematic diagram of an image processing method according to an exemplary embodiment.

[0059] Figure 6The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0060] Figure 7 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0061] Figure 8 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0062] Fig. 9 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0063] Fig.10 is a schematic diagram showing updating an optical flow estimation value according to an exemplary embodiment.

[0064] Fig.11 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0065] Fig.12 The figure is a flowchart of a gradient value processing according to an exemplary embodiment.

[0066] Fig.13 is a schematic diagram showing updating an optical flow estimation value according to an exemplary embodiment.

[0067] Fig.14 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0068] Fig.15 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0070] In order to facilitate the understanding of any embodiment of the present disclosure, first, the optical flow estimation method is described.

[0071] Optical flow estimation methods can be mainly divided into two categories: variational-based methods and convolutional neural network-based methods.

[0072] In one embodiment, under the assumption that continuous images have consistent brightness, temporal continuity and spatial consistency, a variational-based energy equation is proposed to estimate the global optical flow by constraining energy minimization. A small window is used to estimate the local optical flow.

[0073] In one embodiment, in order to handle the discontinuity of optical flow, the quadratic penalty term is replaced by a smooth constraint term, and piecewise smooth processing is allowed.

[0074] In one embodiment, large displacement optical flow is estimated by warping the optical flow in a multi-scale image pyramid.

[0075] In one embodiment, under the assumption that the optical flow field is composed of basic optical flow components, principal component analysis (PCA) is used to calculate the optical flow field.

[0076] In one embodiment, the optical flow is estimated in a local area using image color information and spatial position information.

[0077] In one embodiment, deep learning is introduced into computer vision, and a large number of convolutional neural network methods are used in the field of computer vision.

[0078] Of the two methods mentioned above, the variational method works better when the displacement is small and the brightness change is not large. When there is a large displacement or the image brightness change is large, the estimated optical flow will have a large error. The convolutional neural network method works better on a specific training data set, but when the network is large, the calculation speed is very slow, and when estimating the optical flow for unknown data, there will also be large errors. In summary, no matter which method is used to estimate the optical flow, it is inevitable that the optical flow will be estimated incorrectly.

[0079] like Figure 1 As shown, this embodiment provides an image processing method, the method comprising:

[0080] Step 11, after the image frames are acquired by the image acquisition device, a first optical flow estimation value corresponding to the nth image frame is obtained based on the pixel values ​​of the acquired nth image frame and the n+1th image frame; wherein n is a natural number;

[0081] Step 12: generating an nth predicted frame based on the nth image frame and the first optical flow estimation value;

[0082] Step 13, determining a weight value according to a difference between a pixel value of the nth prediction frame and a pixel value of the n+1th image frame;

[0083] Step 14: Correct the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value.

[0084] Step 15: generating a target prediction frame at the target time domain position based on the collected image frame and the corrected second optical flow estimation value;

[0085] Step 16: Insert the target prediction frame at the target time domain position.

[0086] In one embodiment, the temporal position of the nth image frame may be before or after the temporal position of the (n+1)th image frame.

[0087] In one embodiment, the nth image frame and the (n+1)th image frame may be two frames of images obtained from a video stream captured in real time by a camera.

[0088] In an embodiment, the nth image frame and the (n+1)th image frame may be two images taken continuously by a camera.

[0089] In one embodiment, the acquired nth image frame and the (n+1)th image frame may be stored and acquired by using a direct memory access (DMA) method, so that the processing efficiency of the image frames can be improved.

[0090] In one embodiment, the number of pixels of the nth image frame is the same as that of the n+1th image frame. For example, the number of pixels of the nth image frame and the n+1th image frame is m×n. Here, each pixel corresponds to a pixel value. The pixel value can be between 0 and 255.

[0091] In one embodiment, if the nth image frame and the (n+1)th image frame are grayscale images, the grayscale value of the grayscale image is equal to the pixel value.

[0092] In one embodiment, if the nth image frame and the (n+1)th image frame are color images, the color images need to be converted into grayscale images to obtain the pixel values.

[0093] In one embodiment, the conversion relationship between the gray value Gray of the gray image and the RGB (R, G, B) of the color image can be:

[0094] Gray=0.3R+0.59G+0.11B;

[0095] In one embodiment, assuming that the brightness of the image is the same over continuous time, the first optical flow estimation value can be obtained using a variational optical flow estimation method. The variational optical flow estimation method obtains the first optical flow estimation value corresponding to the nth image frame based on the pixel values ​​of the collected nth image frame and the n+1th image frame. The calculation process is as follows:

[0096] I(x,y,t)=I(x+u,y+v,t+1); Formula (1)

[0097] Among them, x and y are pixel coordinates of pixel points in the image frame; t is time; u represents the horizontal optical flow vector; v represents the vertical optical flow vector; and the first optical flow estimation value is represented by u and v.

[0098] After Taylor expansion of equation (1), we can obtain:

[0099] I x u+I y v+I t =0; (2)

[0100] (2) In the formula, I x ,I y ,I t They are the horizontal gradient, vertical gradient, and time gradient of the image, expressed as vectors:

[0101]

[0102] in,

[0103] When the optical flow is in the normal direction, the direction cannot be estimated, so the optical flow is uncertain, which is the "aperture problem" of the optical flow. To solve this problem, in one embodiment, a smoothness constraint term is introduced and it is assumed that the optical flow field is piecewise smooth. The variational energy equation is used to calculate the optical flows u and v. The energy equation contains two terms, the data term and the smoothness constraint term.

[0104]

[0105] (3) In the formula, the first term is the data term, and the second term is the smoothing constraint term. In order to reduce the influence of local outliers, a convex function is introduced To optimize the equation:

[0106]

[0107]

[0108] For data item E data and the smoothness constraint E smooth Discretize:

[0109]

[0110]

[0111]

[0112] Calculate u and v using the Euler-Lagrange equations:

[0113]

[0114]

[0115] Where n is the number of iterations, is the smoothing coefficient, i and j are pixel positions. Usually, in order to estimate more accurate optical flow, the image pyramid is used to estimate the optical flow at multiple scales, the curved optical flow is used to estimate the large displacement optical flow, and then the multi-scale results are fused to obtain the final first optical flow estimation value.

[0116] In one embodiment, see Figure 2 Based on the collected pixel values ​​of the first image frame a and the second image frame b, a feature map c of the first optical flow estimation value corresponding to the first image frame is obtained.

[0117] In one embodiment, see Figure 3 , is an enlarged view of the local optical flow value of the feature map c, d is the u component of the first optical flow estimation value, and e is the v component corresponding to the first optical flow estimation value. The first optical flow estimation value can be represented by (u, v).

[0118] In one embodiment, see Figure 4 , the first optical flow estimation value corresponding to the nth image frame can also be obtained based on the pixel values ​​of the collected nth image frame and the n+1th image frame based on a convolutional neural network method. The method includes:

[0119] Step a, using a convolutional neural network to extract hierarchical image features to construct the nth image frame feature pyramid and the n+1th frame feature pyramid;

[0120] Step b: Based on the feature pyramid constructed in step a, a U-shaped network structure of the nth image frame is constructed by using a deconvolution operation to perform multi-scale information fusion to obtain multi-scale features;

[0121] Step c, initializing the lowest resolution optical flow field to zero, upsampling the second lowest resolution estimated optical flow field, and performing a deformation operation based on bilinear sampling on the matching features of the n+1th image frame;

[0122] Step d, performing a local similarity calculation based on an inner product on the features of the feature pyramid of the nth image frame and the features of the deformed n+1th image frame obtained in step c, constructing a matching cost, and performing cost aggregation;

[0123] Step e, using the multi-scale features constructed in step b, the optical flow field upsampled in step c, and the matching cost features after cost aggregation in step d as inputs of an optical flow regression network, and estimating a first optical flow estimation value at the resolution;

[0124] Step f: Repeat steps a to e until a first optical flow estimation value at the highest resolution is estimated.

[0125] In one embodiment, an nth predicted frame is generated based on the nth image frame and the first optical flow estimation value, wherein the temporal position of the nth predicted frame in the time domain is the same as the temporal position of the n+1th image frame. Here, the n+1th image frame is the original image frame of the nth predicted frame.

[0126] In one embodiment, the calculation process for obtaining the nth predicted frame may be:

[0127] I2(near(x+f(x)))=I1(x);

[0128] Where I2 represents the nth prediction frame, I1 represents the nth image frame, f(x)∈{u,v} represents the optical flow vector, and near() represents the use of the nearest neighbor's rounded coordinates. The nearest neighbor's rounded coordinates can be the coordinates with the closest distance. For example, if the first coordinate is A(0.8, 0.9), then the second coordinate B(1,1) is the rounded coordinate of the nearest neighbor of the first coordinate.

[0129] In one embodiment, the elements in the pixel value feature vector corresponding to the nth image frame and the elements in the pixel value feature vector corresponding to the (n+1)th image frame are multiplied correspondingly to obtain the pixel value feature vector corresponding to the nth predicted frame.

[0130] In one embodiment, see Figure 5, the first image frame is a 3×3 image frame including 9 pixels, and the corresponding pixel values ​​can be expressed as a feature vector A, A={1, 1, 1, 1, 1, 1, 1, 1}; the second image frame is a 3×3 image frame including 9 pixels, and the corresponding pixel values ​​can be expressed as a feature vector B, B={2, 2, 2, 2, 2, 2, 2, 2}; assuming that after the optical flow estimation operation, the first optical flow estimation value obtained can be expressed as a feature vector C, C={1, 1, 1, 1, 12, 1, 1, 1, 1}. Then the feature vector of the pixel values ​​corresponding to the first predicted frame generated based on the nth image frame and the first optical flow estimation value is a feature vector D, D={1, 1, 1, 1, 12, 1, 1, 1, 1}.

[0131] In one embodiment, the elements in the pixel value feature vector corresponding to the nth prediction frame and the elements in the pixel value feature vector corresponding to the (n+1)th image frame are subtracted from each other to obtain the feature vector of the difference.

[0132] In one embodiment, see again Figure 5 , there is a difference between the feature vector D corresponding to the first predicted frame and the feature vector B corresponding to the second image frame. Subtract the elements in the pixel value feature vector corresponding to the first predicted frame from the elements in the pixel value feature vector corresponding to the second image frame to obtain the feature vector E of the difference, E = {1, 1, 1, 1, 10, 1, 1, 1, 1}; wherein the difference between the pixel values ​​of the 5th pixel is 10, and the difference between the pixel values ​​of other pixels is 1. Here, when the difference between the pixel values ​​of the pixels is greater than the set threshold, the error of the first optical flow estimation value is greater than the error threshold. For example, when the difference between the pixel values ​​of the pixels is greater than 5, the error of the first optical flow estimation value is greater than the error threshold.

[0133] In one embodiment, the feature vector of the weight value is determined according to the feature vector of the difference value, wherein the elements in the feature vector of the weight value correspond to each pixel point one by one, and the weight value is used to adjust the corresponding first optical flow estimation value.

[0134] In one embodiment, adjusting the corresponding first optical flow estimation value may be enlarging or reducing the first optical flow estimation value.

[0135] In one embodiment, when the weight value is used to amplify the first optical flow estimation value, the weight value is greater than A; when the weight value is used to reduce the first optical flow estimation value, the weight value is less than A. Here, A can be 1.

[0136] In one embodiment, see again Figure 5When the difference of the pixel point is less than the set threshold, the weight value of the pixel point is set to a first value, for example, the first value is 1. When the difference of the pixel point is greater than the set threshold, the weight value of the pixel point is set to a second value, for example, the second value is 0. Figure 3 When the difference of the pixel points is less than 5, the weight value of the pixel points is set to 1, and when the difference of the pixel points is greater than 5, the weight value of the pixel points is set to 0. The characteristic vector of the obtained weight value is F, F = {1, 1, 1, 1, 0, 1, 1, 1, 1}.

[0137] In one embodiment, the nth prediction frame I2 and the n+1th image frame I2 may be subjected to exponential difference to establish a weighted signal map c(x). The calculation process is as follows:

[0138]

[0139]

[0140] Among them, max() means taking the maximum value, e is the natural exponent, when the exponential difference between I2 and I2 is close, it has a higher weight, and when the exponential difference between I2 and I2 is large, the weight is small. Here we set a threshold value of 1e -3 , when t is less than the threshold, the weight is zero.

[0141] In one embodiment, an element in a feature vector corresponding to the first optical flow estimation value is multiplied by an element in a feature vector corresponding to the weight value to obtain a corrected feature vector of the second optical flow estimation value.

[0142] In one embodiment, see again Figure 5 , the eigenvector corresponding to the first optical flow estimation value is C, the eigenvector corresponding to the weight value is F, and the eigenvector of the second optical flow estimation value obtained by calculation is G, G = {1, 1, 1, 1, 0, 1, 1, 1, 1}. In this way, the eigenvector G of the corrected second optical flow estimation value and the eigenvector of the nth predicted frame obtained based on the nth image frame is H, H = {1, 1, 1, 1, 0, 1, 1, 1, 1}, and the difference between the pixel values ​​of the pixel points corresponding to the eigenvector H of the nth predicted frame and the eigenvector B of the second image frame is reduced, that is, the error of the corrected second optical flow estimation value is smaller.

[0143] In one embodiment, a dot product may be performed between the weighted confidence map and the first optical flow estimation value to obtain a corrected second optical flow estimation value, and the calculation process is as follows:

[0144] f c (x) = f(x).c(x); where f c Represents the corrected second optical flow estimate.

[0145] In one embodiment, the image acquisition device may acquire a plurality of image frames that are continuous in time domain positions. Here, different image frames correspond to different time domain positions.

[0146] In one embodiment, the image acquisition device acquires a video stream including multiple image frames.

[0147] In one embodiment, a target prediction frame at a predetermined temporal position may be generated based on any one of the multiple image frames and the corrected second optical flow estimation value. Here, the predetermined temporal position is the temporal position in the temporal domain where the multiple image frames acquired by the image acquisition device are located.

[0148] In one embodiment, the target prediction frame is inserted into a time domain position in the time domain where multiple image frames acquired by an image acquisition device are located.

[0149] For example, the image acquisition device acquires a video stream including three image frames that are continuous in time domain position, and the three image frames include image frame a, image frame b and image frame c, wherein the time domain position of image frame a is A, the time domain position of image frame b is B, and the time domain position of image frame c is C. Based on image frame a and the corrected second optical flow estimation value, the generated target prediction frame is prediction frame d, and the time domain position of image frame d is D, and the time domain position D is after the time domain position A and before the time domain position B. Then, the prediction frame d can be inserted between image frame a and image frame b. Then, when the video stream is played continuously, due to the insertion of the prediction frame d, the picture will be clearer and more continuous when the video stream is played.

[0150] In an embodiment of the present disclosure, after an image frame is captured by an image capture device, a first optical flow estimation value corresponding to the nth image frame is obtained based on the pixel values ​​of the captured nth image frame and the n+1th image frame, and an nth prediction frame is generated based on the nth image frame and the first optical flow estimation value. Here, since the nth prediction frame is generated based on the nth image frame and the first optical flow estimation value, the n+1th image frame is the original image frame of the nth prediction frame. A weight value is determined based on the difference between the pixel values ​​of the nth prediction frame and the pixel values ​​of the n+1th image frame; the first optical flow estimation value is corrected based on the weight value to obtain a corrected second optical flow estimation value; a target prediction frame at a target time domain position is generated based on the captured image frame and the corrected second optical flow estimation value; and the target prediction frame is inserted at the target time domain position. In this way, the weight value can be determined according to the difference between the pixel value of the nth predicted frame and the pixel value of the original image frame, i.e., the n+1th image frame, and the first optical flow estimation value can be corrected using the weight value to obtain the corrected second optical flow estimation value. The corrected second optical flow estimation value can then be used to generate a predicted frame. Compared with the method of directly using the uncorrected first optical flow estimation value to generate a predicted frame without correcting the first optical flow estimation value using the weight value, the error of the first optical flow estimation value can be reduced, making the generated predicted frame more accurate.

[0151] In one embodiment, the difference value is negatively correlated with the weight value determined according to the difference value.

[0152] In one embodiment, when the difference is greater than a set threshold, the weight value determined according to the difference is less than the weight threshold.

[0153] In one embodiment, the difference is inversely proportional to the weight value determined according to the difference.

[0154] like Figure 6 As shown, this embodiment provides an image processing method, step 13, determining the weight value according to the difference between the pixel value of the nth prediction frame and the pixel value of the n+1th image frame, includes:

[0155] Step 61: If the difference is greater than a set threshold, determine that the weight value is less than the weight threshold.

[0156] In one embodiment, when the required error of the first optical flow estimation value is less than an error threshold, the set threshold is less than A; when the required error of the first optical flow estimation value is greater than the error threshold, the set threshold is greater than A. In this way, the set threshold can be adaptively adjusted according to the required error of the first optical flow estimation value.

[0157] like Figure 7As shown, this embodiment provides an image processing method, step 14, correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value, including:

[0158] Step 71: Obtain a corrected second optical flow estimation value based on the product of the weight value and the first optical flow estimation value.

[0159] In one embodiment, an element in a feature vector corresponding to the first optical flow estimation value is multiplied by an element in a feature vector corresponding to the weight value to obtain a corrected feature vector of the second optical flow estimation value.

[0160] In one embodiment, see again Figure 5 , the eigenvector corresponding to the first optical flow estimation value is C, the eigenvector corresponding to the weight value is F, and the elements in the eigenvector corresponding to the first optical flow estimation value are multiplied by the elements in the eigenvector corresponding to the weight value, and the eigenvector G of the corrected second optical flow estimation value is calculated, G = {1, 1, 1, 1, 0, 1, 1, 1, 1}. In this way, the eigenvector G of the corrected second optical flow estimation value and the eigenvector of the nth predicted frame obtained based on the nth image frame is H, H = {1, 1, 1, 1, 0, 1, 1, 1, 1}, and the difference between the pixel values ​​of the pixel points corresponding to the eigenvector H of the nth predicted frame and the eigenvector B of the second image frame is reduced, that is, the error of the corrected second optical flow estimation value is smaller.

[0161] like Figure 8 As shown, this embodiment provides an image processing method, the method further comprising:

[0162] Step 81: If the corrected second optical flow estimation value is less than an estimation value threshold, update the second optical flow estimation value that is less than the estimation value threshold to a preset estimation value.

[0163] In one embodiment, the preset estimated value may be an average value of the second optical flow estimated values ​​corresponding to all pixels in the image frame.

[0164] In an embodiment, the preset estimated value may be an average value of the second optical flow estimated values ​​corresponding to all pixels in a certain pixel area in the image frame.

[0165] In one embodiment, the estimation value threshold may be determined according to the relative motion speed between the terminal capturing the image frame and the captured target object.

[0166] In one embodiment, when the relative motion speed between the terminal that captures the image frame and the captured target object is greater than a speed threshold, the estimated value threshold is greater than B; when the relative motion speed between the terminal that captures the image frame and the captured target object is less than the speed threshold, the estimated value threshold is less than B. In this way, the estimated value threshold can be adjusted according to the relative motion speed between the terminal that captures the image frame and the captured target object, so that the corrected second optical flow estimated value obtained is more consistent with the actual scene.

[0167] In one embodiment, see again Figure 5 The eigenvector of the corrected second optical flow estimation value is G, G = {1, 1, 1, 1, 0, 1, 1, 1, 1}. If the second optical flow estimation value of the fifth element is 0 which is less than the estimation value threshold 1, 0 can be replaced by a preset estimation value 1 / 3.

[0168] like Fig. 9 As shown, this embodiment provides an image processing method, the method further comprising:

[0169] Step 91, generating a second optical flow estimation value matrix according to the second optical flow estimation values ​​of pixels in the nth image frame or the (n+1)th image frame;

[0170] Step 81, if the corrected second optical flow estimation value is less than an estimation value threshold, updating the second optical flow estimation value less than the estimation value threshold to a preset estimation value, comprises:

[0171] Step 92: If the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, update the second optical flow estimation value of the Nth element to: the average value between adjacent elements of the Nth element; N is a natural number.

[0172] In one embodiment, when the nth image frame or the n+1th image frame is an m×n image frame, the second optical flow estimation value matrix is ​​an m×n matrix, and each pixel point in the image frame corresponds to an element in the second optical flow estimation value matrix, where m and n are positive integers.

[0173] In one embodiment, see Fig.10 , the adjacent elements of the Nth element can be Fig.10 Elements in the top, bottom, left, and right positions. Can also be Fig.10 Elements at the top, bottom, left, right, top-left, top-right, bottom-left, and bottom-right positions.

[0174] like Fig.11 As shown, this embodiment provides an image processing method, the method further comprising:

[0175] Step 111: Obtain an optical flow estimation gradient value of the second optical flow estimation value;

[0176] Step 112: superimpose the optical flow estimation gradient value and the second optical flow estimation value to obtain a superimposed second optical flow estimation value;

[0177] Step 113: input the superimposed second optical flow estimation value into a global energy functional; wherein the global energy functional includes a data term and a smoothing term; solve the global energy functional to obtain an updated second optical flow estimation value.

[0178] In an embodiment, the optical flow estimation gradient value of the second optical flow estimation value may be a gradient value of a setting direction. Here, the setting direction may be an X direction of the pixel matrix or a Y direction of the pixel matrix.

[0179] In one embodiment, the optical flow estimation gradient value is the difference between the pixel values ​​of any two adjacent pixels in the set direction.

[0180] In one embodiment, see Fig.12 The second optical flow estimation values ​​corresponding to the four pixel points in the X direction are 100, 90, 90 and 90 respectively, and the corresponding optical flow estimation gradient values ​​are 10, 0, 0 and 0 respectively. The optical flow estimation gradient value is superimposed with the second optical flow estimation value, and the superimposed second optical flow estimation values ​​are 110, 90, 90 and 90 respectively.

[0181] In one embodiment, the variational energy equation is used to calculate the optical flows u and v. The global energy function contains two terms, the data term E data With smoothing term The energy equation is:

[0182]

[0183] The global energy functional is solved to obtain an updated second optical flow estimate.

[0184] In order to further understand the technical solution of the embodiment of the present disclosure, the embodiment of the present disclosure is further described below by an example:

[0185] Example 1:

[0186] Step 1: Based on the pixel values ​​of the first image frame and the second image frame collected, obtain a first optical flow estimation value corresponding to the first image frame. The first optical flow estimation value can be obtained by any optical flow estimation method, for example, the optical flow can be estimated by a method based on a convolutional neural network, or by a variational method. The first optical flow estimation value obtained by any method is a two-layer matrix, in which one layer represents u as a horizontal motion vector, and the other layer represents v as a vertical motion vector.

[0187] Step 2: Generate a first prediction frame based on the first image frame and the first optical flow estimation value. The calculation process for obtaining the first prediction frame is:

[0188] I2(near(x+f(x)))=I1(x);

[0189] Where I2 represents the first prediction frame, I1 represents the first image frame, f(x)∈{u,v} represents the optical flow vector, and near() represents the use of the nearest neighbor's rounded coordinates. The nearest neighbor's rounded coordinates can be the coordinates with the closest distance. For example, if the first coordinate is A(0.8, 0.9), then the second coordinate B(1,1) is the rounded coordinate of the nearest neighbor of the first coordinate.

[0190] Step 3: Determine the weight value according to the difference between the pixel value of the first prediction frame and the pixel value of the second image frame. Make an exponential difference between the first prediction frame I2 and the second image frame I2 to establish a weighted signal map c(x). The calculation process is as follows:

[0191]

[0192]

[0193] Among them, max() means taking the maximum value, e is the natural exponent, when the exponential difference between I2 and I2 is close, it has a higher weight, and when the exponential difference between I2 and I2 is large, the weight is small. Here we set a threshold value of 1e -3 , when t is less than the threshold, the weight is zero.

[0194] Step 4: Correct the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value. Perform a dot product of the weighted signal map and the first optical flow estimation value to obtain a corrected second optical flow estimation value. The calculation process is as follows:

[0195] f c (x) = f(x).c(x); where f c Represents the corrected second optical flow estimate.

[0196] When the value of I2 differs greatly from I2 at a certain point, because the weight is 0, the position of the point on the corrected optical flow is 0, which is a hole.

[0197] Step 5: If the corrected second optical flow estimation value is less than the estimation value threshold, the second optical flow estimation value less than the estimation value threshold is updated to a preset estimation value. For the holes in the corrected optical flow (corresponding to the pixel points of the second optical flow estimation value less than the estimation value threshold), a soft fusion method is used to fill the positions of the holes. Fig.13 .

[0198] Fig.13 The black squares in the figure represent holes, and the arrows indicate that non-zero values ​​are searched from the top, bottom, left, and right directions of the holes, and then the non-zero values ​​found are fused. The fusion method is obtained by calculating the average value (here, the average value is a preset estimated value):

[0199]

[0200] Among them, z(x) is the value after the hole is filled, v i∈{up,down,left,right} Represents non-zero values ​​in the four directions of up, down, left and right. Divisor represents the number of non-zero values ​​in the four directions of up, down, left and right. The optical flow filled with soft fusion is more natural than the optical flow filled directly with the nearest neighbor method.

[0201] Step 6: Obtain the optical flow estimation gradient value of the second optical flow estimation value; superimpose the optical flow estimation gradient value with the second optical flow estimation value to obtain the superimposed second optical flow estimation value; input the superimposed second optical flow estimation value into the global energy function; wherein the global energy function includes a data term and a smoothing term; solve the global energy function to obtain an updated second optical flow estimation value. The obtained global energy function is:

[0202]

[0203] in, Represents the gradient.

[0204] like Fig.14 As shown, in this embodiment, an image processing device is provided, which includes a calculation module 141, a generation module 142, a determination module 143, a correction module 144 and an insertion module 145; wherein,

[0205] The calculation module 141 is used to: after the image frame is acquired by the image acquisition device, based on the pixel values ​​of the acquired nth image frame and the (n+1)th image frame, obtain a first optical flow estimation value corresponding to the nth image frame; wherein n is a natural number;

[0206] The generating module 142 is configured to: generate an nth predicted frame based on the nth image frame and the first optical flow estimation value;

[0207] The determination module 143 is used to determine a weight value according to a difference between a pixel value of the nth predicted frame and a pixel value of the n+1th frame image;

[0208] The correction module 144 is used to: correct the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value;

[0209] The generating module 142 is further configured to: generate a target prediction frame at a target temporal position based on the collected image frame and the second optical flow estimation value;

[0210] The inserting module 145 is used to insert the target prediction frame at the target time domain position.

[0211] In one embodiment, the determination module 143 is further configured to configure: the difference is negatively correlated with the weight value determined according to the difference.

[0212] In one embodiment, the determining module 143 is further configured to:

[0213] If the difference is greater than a set threshold, it is determined that the weight value is less than the weight threshold.

[0214] In one embodiment, the correction module 144 is further configured to:

[0215] Based on the product of the weight value and the first optical flow estimation value, a corrected second optical flow estimation value is obtained.

[0216] In one embodiment, the apparatus further includes an updating module 146, wherein the updating module 146 is further configured to:

[0217] If the corrected second optical flow estimation value is less than an estimation value threshold, the second optical flow estimation value less than the estimation value threshold is updated to a preset estimation value.

[0218] In one embodiment,

[0219] The generating module 142 is further used for:

[0220] Generate a second optical flow estimation value matrix according to second optical flow estimation values ​​of pixels in the nth image frame or the (n+1)th image frame;

[0221] The correction module is also used to: if the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, update the second optical flow estimation value of the Nth element to: the average value between adjacent elements of the Nth element; N is a natural number.

[0222] In one embodiment, the device further comprises an acquisition module 147, wherein:

[0223] The acquisition module 147 is used to obtain the optical flow estimation gradient value of the second optical flow estimation value; superimpose the optical flow estimation gradient value with the second optical flow estimation value to obtain the superimposed second optical flow estimation value; input the superimposed second optical flow estimation value into the global energy function; wherein the global energy function includes a data term and a smoothing term; solve the global energy function to obtain the updated second optical flow estimation value.

[0224] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0225] The present disclosure also provides a communication device, including:

[0226] antenna;

[0227] Memory;

[0228] The processor is connected to the antenna and the memory respectively, and is used to control the antenna to receive and send wireless signals by executing an executable program stored in the memory, and can execute the steps of the wireless network access method provided in any of the above embodiments.

[0229] The communication device provided in this embodiment may be the aforementioned terminal or base station. The terminal may be various types of person-mounted terminals or vehicle-mounted terminals. The base station may be various types of base stations, for example, a 4G base station or a 5G base station.

[0230] The antenna may be various types of antennas, for example, a mobile antenna such as a 3G antenna, a 4G antenna or a 5G antenna; the antenna may also include: a WiFi antenna or a wireless charging antenna, etc.

[0231] The memory may include various types of storage media, which are non-transitory computer storage media that can continue to remember information stored thereon after the communication device loses power.

[0232] The processor can be connected to the antenna and the memory through a bus, etc., and is used to read the executable program stored in the memory, for example, at least one of the methods shown in any embodiment of the present disclosure.

[0233] An embodiment of the present disclosure also provides a non-temporary computer-readable storage medium, which stores an executable program, wherein when the executable program is executed by a processor, the steps of the wireless network access method provided in any of the aforementioned embodiments are implemented, for example, at least one of the methods shown in any embodiment of the present disclosure.

[0234] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0235] Fig.15 is a block diagram of an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0236] Reference Fig.15 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0237] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0238] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0239] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0240] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0241] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0242] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0243] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0244] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0245] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0246] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by the processor 820 of the electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0247] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0248] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: After the image frames are captured by the image capture device, a first optical flow estimation value corresponding to the nth image frame is obtained based on the pixel values ​​of the captured nth image frame and the n+1th image frame; wherein n is a natural number; Generate an nth predicted frame based on the nth image frame and the first optical flow estimation value; Determine a weight value according to a difference between a pixel value of the nth predicted frame and a pixel value of the n+1th frame image; wherein the difference is negatively correlated with the weight value; If the difference is less than a set threshold, determining the weight value to be a first value; If the difference is greater than the set threshold, determining the weight value to be a second value; Correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value; Generate a second optical flow estimation value matrix according to second optical flow estimation values ​​of pixels in the nth image frame or the (n+1)th image frame; If the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, updating the second optical flow estimation value of the Nth element to: an average value between adjacent elements of the Nth element, where N is a natural number; Generate a target prediction frame at the target time domain position based on the collected image frame and the second optical flow estimation value; The target prediction frame is inserted at the target time domain position.

2. The method according to claim 1, characterized in that: The step of determining a weight value according to a difference between a pixel value of the nth prediction frame and a pixel value of the n+1th image frame comprises: If the difference is greater than a set threshold, it is determined that the weight value is less than the weight threshold.

3. The method according to claim 1, characterized in that The step of correcting the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value includes: Based on the product of the weight value and the first optical flow estimation value, a corrected second optical flow estimation value is obtained.

4. The method according to claim 1, characterized in that The method further comprises: Obtain an optical flow estimation gradient value of the second optical flow estimation value; Superimposing the optical flow estimation gradient value and the second optical flow estimation value to obtain a superimposed second optical flow estimation value; Inputting the superimposed second optical flow estimation value into a global energy function; wherein the global energy function includes a data term and a smoothing term; Solve the global energy functional to obtain an updated second optical flow estimate.

5. An image processing device, characterized in that: The device comprises a calculation module, a generation module, a determination module, a correction module, an update module and an insertion module; wherein, The calculation module is used to: after the image frame is acquired by the image acquisition device, based on the pixel values ​​of the acquired nth image frame and the (n+1)th image frame, obtain a first optical flow estimation value corresponding to the nth image frame; wherein n is a natural number; The generating module is used to: generate an nth predicted frame based on the nth image frame and the first optical flow estimation value; The determination module is used to: determine a weight value according to a difference between a pixel value of the nth predicted frame and a pixel value of the n+1th frame image; wherein the difference is negatively correlated with the weight value; The determination module is further configured to: if the difference is less than a set threshold, determine that the weight value is a first value; If the difference is greater than the set threshold, determining the weight value to be a second value; The correction module is used to: correct the first optical flow estimation value based on the weight value to obtain a corrected second optical flow estimation value; The generating module is further used to: generate a second optical flow estimation value matrix according to the second optical flow estimation values ​​of pixels in the nth image frame or the n+1th image frame; The updating module is configured to: if the second optical flow estimation value of the Nth element in the second optical flow estimation value matrix is ​​less than the estimation value threshold, update the second optical flow estimation value of the Nth element to: an average value between adjacent elements of the Nth element; wherein N is a natural number; The generating module is further used to: generate a target prediction frame at a target time domain position based on the collected image frame and the second optical flow estimation value; The inserting module is used to insert the target prediction frame at the target time domain position.

6. The device according to claim 5, characterized in that The determining module is further used for: If the difference is greater than a set threshold, it is determined that the weight value is less than the weight threshold.

7. The device according to claim 5, characterized in that The correction module is further used for: Based on the product of the weight value and the first optical flow estimation value, a corrected second optical flow estimation value is obtained.

8. The device according to claim 5, characterized in that The device also includes an acquisition module, wherein: The acquisition module is used to obtain the optical flow estimation gradient value of the second optical flow estimation value; superimpose the optical flow estimation gradient value with the second optical flow estimation value to obtain the superimposed second optical flow estimation value; input the superimposed second optical flow estimation value into the global energy function; wherein the global energy function includes a data term and a smoothing term; solve the global energy function to obtain the updated second optical flow estimation value.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing a computer service that can be run on the processor, wherein the processor implements the image processing method according to any one of claims 1 to 4 when running the computer service.

10. A storage medium, characterized in that: The storage medium contains computer executable instructions, and the computer executable instructions are executed by a processor to implement the image processing method according to any one of claims 1 to 4.

Images