Optical flow acquisition method and device

Abstract

The present application discloses an optical flow acquisition method and device, which relates to the field of video image processing and is used to reduce the time consumption of iterative calculation in the process of calculating optical flow. The optical flow acquisition method includes: determining a first similarity between a first pixel block in the i-1th frame image and a second pixel block in the i-th frame image; performing at least one of the following two processes: determining a second similarity between the first pixel block and a third pixel block in the i-th frame image, the coordinates of the third pixel block being obtained by the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame; determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, the coordinates of the fourth pixel block being obtained by the optical flow of the adjacent pixel blocks of the first pixel block; obtaining a target optical flow of the first pixel block from the i-1th frame image to the i-th frame image according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block.

Description

Technical Field

[0001] The present application relates to the field of video image processing, and in particular to a method and device for acquiring optical flow. Background Art

[0002] Optical flow is a concept in the detection of object motion in the field of view, which is used to describe the motion of the observed target, surface or edge caused by the motion relative to the observer. The optical flow algorithm is a method that detects the intensity of image pixels over time and then infers the speed and direction of the object. It is widely used in image processing and can be used for motion detection, motion tracking, etc.

[0003] The essence of the optical flow algorithm is to iteratively solve the optimization problem. With the brightness consistency assumption and motion consistency assumption as the core ideas, the corresponding energy function equation is constructed. Then, through a large number of iterative calculations, the optimal solution of the energy function equation is gradually approached. In the iterative calculation process, the initial value used is not controlled, which makes the iterative calculation time-consuming. Summary of the invention

[0004] The embodiments of the present application provide an optical flow acquisition method and device for reducing the time consumption of iterative calculation in the process of calculating optical flow.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for acquiring an optical flow is provided, comprising: determining a first similarity between a first pixel block in an i-1th frame image and a second pixel block in an i-th frame image, wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer; performing at least one of the following two processes: determining a second similarity between the first pixel block and a third pixel block in the i-th frame image, wherein the coordinates of the third pixel block are obtained by a historical optical flow of the first pixel block from an i-2th frame image to an i-1th frame; determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, wherein the coordinates of the fourth pixel block are obtained by an optical flow of adjacent pixel blocks of the first pixel block; and obtaining a target optical flow of the first pixel block from the i-1th frame image to the i-th frame image based on at least one of the second similarity and the third similarity, the first similarity, and gradient information of the first pixel block.

[0007] The optical flow acquisition method provided by the present application determines the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image, and the coordinates of the second pixel block are the same as the coordinates of the first pixel block, which reflects the consistency of the first pixel block remaining motionless. By determining the second similarity between the first pixel block in the i-1th frame image and the third pixel block in the i-th frame image, the coordinates of the third pixel block are obtained by the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame, which reflects the temporal consistency between the optical flow of the first pixel block from the i-2th frame image to the i-1th frame and the optical flow from the i-1th frame image to the i-th frame image. By determining the third similarity between the first pixel block and the fourth pixel block in the i-th frame image, the coordinates of the fourth pixel block are obtained by the optical flow of the adjacent pixel blocks of the first pixel block, which reflects the spatial consistency between the optical flow of the first pixel block and the optical flow of the adjacent pixel blocks. The above similarity is used to optimize the initial value of the iteratively calculated optical flow, thereby reducing the number of iterations. A lightweight optical flow algorithm can be implemented, which can converge quickly at a low number of iterations.

[0008] In a possible implementation, the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame. The size of the third pixel block is the same as that of the first pixel block.

[0009] In a possible implementation, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of the adjacent pixel blocks of the first pixel block. The size of the fourth pixel block is the same as that of the first pixel block.

[0010] In a possible implementation, according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block, the target optical flow of the first pixel block from the i-1th frame image to the i-th frame image is obtained, including: determining the highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and selecting the optical flow corresponding to the highest similarity as the initial value of the optical flow; combining the gradient information of the first pixel block, performing an approximate Gauss-Newton gradient descent iteration to solve the initial value of the optical flow, and using the iterative result obtained when exiting because the exit condition is met as the target optical flow. That is, optimizing the initial value of the optical flow according to the highest similarity can reduce the number of iterations.

[0011] In a possible implementation, according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block, a target optical flow of the first pixel block from the i-1th frame image to the i-th frame image is obtained, including: determining at least two highest similarities according to at least one of the second similarity and the third similarity and the first similarity, and selecting optical flows corresponding to the at least two highest similarities as at least two initial optical flow values; combining the gradient information of the first pixel block, performing approximate Gauss-Newton gradient descent iteration on the at least two initial optical flow values, and setting the exit condition to be the same, and when exiting due to satisfying the exit condition, selecting the iteration result with the smallest energy function as the target optical flow. That is, the initial optical flow values ​​corresponding to multiple highest similarities are preferred, so as to further optimize the optical flow according to the iteration result.

[0012] In a possible implementation, the gradient information includes a sum of gradient values ​​in an X direction, a sum of squares of gradient values ​​in an X direction, a sum of gradient values ​​in a Y direction, a sum of squares of gradient values ​​in a Y direction, and a sum of products of gradient values ​​in an X direction and a Y direction. The method further includes: performing convolution on each pixel point of the first pixel block and an X-direction Sobel operator to obtain an X-direction gradient matrix, performing convolution on each pixel point of the first pixel block and a Y-direction Sobel operator to obtain a Y-direction gradient matrix; accumulating and summing all gradient values ​​of the X-direction gradient matrix to obtain a sum of gradient values ​​in an X direction, squaring each gradient value of the X-direction gradient matrix and then accumulating and summing the sum of squares of gradient values ​​in an X direction; accumulating and summing all gradient values ​​of the Y-direction gradient matrix to obtain a sum of gradient values ​​in an Y direction, squaring each gradient value of the Y-direction gradient matrix and then accumulating and summing the sum of squares of gradient values ​​in an Y direction; and multiplying gradient values ​​at the same position of the X-direction gradient matrix and the Y-direction gradient matrix and then accumulating and summing the gradient values ​​in an X-direction and a Y-direction.

[0013] In a possible implementation, the method further includes: determining a fourth similarity between the first pixel block and the fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained by the target optical flow; and determining the confidence of the target optical flow based on the first similarity and the fourth similarity. Limited by the principle of the gradient-based optical flow algorithm, large errors are easily caused in gradient flat areas. The present application also adds a confidence mechanism, where the confidence of the optical flow with a large error is low, and the confidence of the optical flow with a small error is high. Users can select an optical flow with a high confidence.

[0014] In a possible implementation, the confidence of the target optical flow is determined based on the first similarity and the fourth similarity, including: if the ratio of the value of the fourth similarity to the value of the first similarity is greater than a first threshold, the target optical flow is determined to be low confidence; otherwise, if the value of the fourth similarity is greater than a second threshold, the target optical flow is determined to be low confidence; otherwise, the target optical flow is determined to be high confidence.

[0015] In a possible implementation, the method further includes: inserting an intermediate image between the i-1th frame image and the ith frame image according to the target optical flow. Due to the computing power limitation of the processor, there is a gap between the image rendering rate and the display frame rate. The time domain super-resolution interpolation function can be realized with the help of the result of the optical flow acquisition method of the present application.

[0016] In a second aspect, an optical flow acquisition device is provided, including: a determination module, used to determine a first similarity between a first pixel block in an i-1th frame image and a second pixel block in an i-th frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer; the determination module, further used to perform at least one of the following two processes: determining a second similarity between the first pixel block and a third pixel block in the i-th frame image, wherein the coordinates of the third pixel block are obtained by a historical optical flow of the first pixel block from an i-2th frame image to an i-1th frame; determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, wherein the coordinates of the fourth pixel block are obtained by an optical flow of adjacent pixel blocks of the first pixel block; an acquisition module, used to obtain a target optical flow of the first pixel block from the i-1th frame image to the i-th frame image based on at least one of the second similarity and the third similarity, the first similarity, and gradient information of the first pixel block.

[0017] In a possible implementation, the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame.

[0018] In a possible implementation, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of the adjacent pixel blocks of the first pixel block.

[0019] In a possible implementation, the acquisition module is specifically used to: determine the highest similarity based on at least one of the second similarity and the third similarity and the first similarity, and select the optical flow corresponding to the highest similarity as the initial optical flow value; combine the gradient information of the first pixel block, perform an approximate Gauss-Newton gradient descent iterative solution on the initial optical flow value, and use the iterative result obtained when exiting because the exit condition is met as the target optical flow.

[0020] In a possible implementation, the acquisition module is specifically used to: determine at least two highest similarities based on at least one of the second similarity and the third similarity and the first similarity, and select optical flows corresponding to the at least two highest similarities as at least two optical flow initial values; in combination with the gradient information of the first pixel block, perform approximate Gauss-Newton gradient descent iterative solutions on the at least two optical flow initial values, and set the exit condition to be the same. When exiting due to satisfying the exit condition, select the iterative result with the smallest energy function as the target optical flow.

[0021] In a possible implementation, the gradient information includes the sum of the gradient values ​​in the X direction, the sum of the squares of the gradient values ​​in the X direction, the sum of the gradient values ​​in the Y direction, the sum of the squares of the gradient values ​​in the Y direction, and the sum of the products of the gradient values ​​in the X direction and the Y direction. The acquisition module is further used to: perform convolution on each pixel point of the first pixel block and the Sobel operator in the X direction to obtain the gradient matrix in the X direction, perform convolution on each pixel point of the first pixel block and the Sobel operator in the Y direction to obtain the gradient matrix in the Y direction; perform cumulative summation on all the gradient values ​​of the gradient matrix in the X direction to obtain the sum of the gradient values ​​in the X direction, perform cumulative summation on each gradient value of the gradient matrix in the X direction to obtain the sum of the squares of the gradient values ​​in the X direction; perform cumulative summation on all the gradient values ​​of the gradient matrix in the Y direction to obtain the sum of the gradient values ​​in the Y direction, perform cumulative summation on each gradient value of the gradient matrix in the Y direction to obtain the sum of the squares of the gradient values ​​in the Y direction; perform cumulative summation on the gradient values ​​at the same position of the gradient matrix in the X direction and the gradient matrix in the Y direction to obtain the sum of the products of the gradient values ​​in the X direction and the Y direction.

[0022] In a possible implementation, the determination module is further used to: determine a fourth similarity between the first pixel block and the fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained by the target optical flow; and determine the confidence of the target optical flow based on the first similarity and the fourth similarity.

[0023] In a possible implementation, the determination module is specifically used to: if the ratio of the value of the fourth similarity to the value of the first similarity is greater than a first threshold, determine that the target optical flow is low confidence; otherwise, if the value of the fourth similarity is greater than a second threshold, determine that the target optical flow is low confidence; otherwise, determine that the target optical flow is high confidence.

[0024] In a possible implementation, the optical flow acquisition device further includes a frame insertion module, which is used to insert an intermediate image between the (i-1)th frame image and the i-th frame image according to the target optical flow.

[0025] In a third aspect, an optical flow acquisition device is provided, comprising a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method of the first aspect and possible implementation methods thereof.

[0026] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium are executed on a computer or a processor, the computer or the processor executes the method of the first aspect and its possible implementation methods.

[0027] In a fifth aspect, a computer program product comprising instructions is provided, which, when the instructions are executed on a computer or a processor, enables the computer or the processor to execute the method of the first aspect and its possible implementation methods.

[0028] The technical effects of the second to fifth aspects can refer to the contents described in various possible implementation modes of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a flow chart of an optical flow acquisition method provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a pixel block provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of an optical flow provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a second pixel block provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a third pixel block provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of adjacent pixel blocks of a first pixel block provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of a fourth pixel block provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of another fourth pixel block provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of a Sobel operator provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of an approximate Gauss-Newton gradient descent iterative solution provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram of another optical flow acquisition method provided in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of a flow chart of another optical flow acquisition method provided in an embodiment of the present application;

[0041] Fig.13 A schematic diagram of a flow chart of another optical flow acquisition method provided in an embodiment of the present application;

[0042] Fig.14 A schematic diagram of a flow chart of another optical flow acquisition method provided in an embodiment of the present application;

[0043] Fig.15 A schematic diagram of the structure of an optical flow acquisition device provided in an embodiment of the present application;

[0044] Fig.16 A schematic diagram of the structure of another optical flow acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In the field of video image processing, especially for terminal products, high requirements are placed on the performance and effect of optical flow algorithms. It is usually necessary to obtain full-image optical flow information with high accuracy and good stability at high speed. Therefore, implementing lightweight optical flow algorithms through integrated circuits is a better solution. However, the traditional optical flow algorithm requires a large number of iterative calculations, which limits the improvement of the frame rate. In addition, lightweight optical flow algorithms usually have large errors in flat gradient areas and poor stability.

[0046] For example, for a large number of iterative calculations, each image block is calculated independently, and the initial value used is not controlled. If a poor initial value is used, a large number of iterative calculations are required to gradually approach a better solution. For poor stability, the optical flow of two adjacent frames is calculated independently of the two frames, and the correlation between the two frames is not fully utilized. For large errors in the flat gradient area, due to the constraints of the optical flow algorithm principle, the flat gradient area is not easy to converge, and the error is usually large.

[0047] The present application provides an optical flow acquisition method and device, which determines the first similarity between the first pixel block in the i-1th frame image and the second pixel block in the i-th frame image, and the coordinates of the second pixel block are the same as the coordinates of the first pixel block, which reflects the consistency of the first pixel block remaining motionless. By determining the second similarity between the first pixel block in the i-1th frame image and the third pixel block in the i-th frame image, the coordinates of the third pixel block are obtained from the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame, which reflects the temporal consistency between the optical flow of the first pixel block from the i-2th frame image to the i-1th frame and the optical flow from the i-1th frame image to the i-th frame image. By determining the third similarity between the first pixel block and the fourth pixel block in the i-th frame image, the coordinates of the fourth pixel block are obtained from the optical flow of the adjacent pixel blocks of the first pixel block, which reflects the spatial consistency between the optical flow of the first pixel block and the optical flow of the adjacent pixel blocks. The above similarity is used to optimize the initial value of the iteratively calculated optical flow, thereby reducing the number of iterations. A lightweight optical flow algorithm can be implemented, which can converge quickly at a low number of iterations.

[0048] like Figure 1As shown, an embodiment of the present application provides an optical flow acquisition method, including steps S101-S104, wherein at least one of steps S102 and S103 is performed.

[0049] S101 . Determine a first similarity between a first pixel block in an (i-1)th frame image and a second pixel block in an (i)th frame image.

[0050] In the embodiment of the present application, the i-th frame image is the current frame image in the video stream, the i-1-th frame image refers to the previous frame image (also referred to as the reference frame image), the i-2-th frame image refers to the previous two frames image, and so on. i is a positive integer.

[0051] The image is gridded to obtain pixel blocks (also called unit blocks), each pixel block includes at least four pixels (i.e., 2 pixels * 2 pixels), the pixel block can be rectangular or square, and the size of the pixel block is configurable, for example, 6 pixels * 6 pixels, 8 pixels * 8 pixels, 10 pixels * 10 pixels, 12 pixels * 12 pixels, 14 pixels * 14 pixels, etc.

[0052] Adjacent pixel blocks include at least one common pixel. Figure 2 As shown, taking the size of each pixel block as 3 pixels * 3 pixels as an example, pixel blocks A1 and A4, A1 and A2, A1 and A3, A2 and A3, A2 and A4, and A3 and A4 all include common pixels. The significance of adjacent pixel blocks including common pixels is that if the common pixels belong to multiple pixel blocks, it can be considered that the calculation results of the pixel area composed of the common pixels can be obtained by weighted average according to the calculation results of the pixel blocks to which they belong, which can better ensure robustness. If there is no overlap between pixel blocks, the calculation results of any pixel area are only independently determined by the calculation results of a single pixel block, so there is a risk of large individual deviations.

[0053] like Figure 2 As shown in the figure, if a plane coordinate system is established for an image, the pixel point in the upper left corner of the image is generally used as the origin, the right is the X direction, and the downward is the Y direction. When an object is moving, the brightness pattern of the corresponding points on the adjacent images is also moving. This apparent motion that characterizes the brightness pattern of the image is called optical flow. Specifically, optical flow represents the speed and direction of movement of each pixel in two adjacent frames of images. For example, Figure 3 As shown, assuming that the optical flow of a target (for example, the first pixel block) in the i-1th frame image is (2, 3), it means that the target (for example, the first pixel block) in the i-1th frame image moves 2 pixels to the right and 3 pixels downward in the i-th frame image. Therefore, the optical flow (2, 3) reflects the displacement of the same target in two adjacent frame images.

[0054] In the embodiment of the present application, the coordinates of the pixel point at the upper left corner of the pixel block are used as the coordinates of the pixel block. Of course, the coordinates of the pixel point at the center of the pixel block can also be used as the coordinates of the pixel block, which is not limited in the present application.

[0055] When calculating the optical flow, the present application exemplarily starts from the pixel block in the upper left corner, calculates one row of pixel blocks to the right, and then continues to calculate from the leftmost pixel block in the next row to the right. Figure 2 For example, the calculation order of the pixel blocks is A1, A2, A3, A4.

[0056] The coordinates of the second pixel block in the i-th frame image (e.g., the coordinates of the pixel point in the upper left corner) are the same as the coordinates of the first pixel block in the i-1-th frame image (e.g., the coordinates of the pixel point in the upper left corner), and the size of the second pixel block is the same as the size of the first pixel block. In other words, the coordinates of the first pixel block (e.g., the coordinates of the pixel point in the upper left corner) are used as the coordinates of the second pixel block in the i-th frame image (e.g., the coordinates of the pixel point in the upper left corner), and based on the coordinates, an area of ​​the same size as the first pixel block is selected in the i-th frame image as the second pixel block.

[0057] The significance of this step is that the target may not move in the two frames, that is, the pixel blocks at the same position in the two frames may be the same, and the corresponding optical flow is (0, 0).

[0058] For example, Figure 4 As shown, taking the size of the first pixel block as 3 pixels * 3 pixels as an example, the coordinates of the first pixel block in the i-1th frame image are (1, 2), then the coordinates of the second pixel block in the i-th frame image are (1, 2), and the size of the second pixel block is the same as that of the first pixel block, both of which are 3 pixels * 3 pixels.

[0059] In image processing, the indicators for evaluating the similarity between images include but are not limited to the sum of absolute difference (SAD), the sum of square difference (SSD), the normalized cross correlation (NCC), etc., which are not limited in this application. For SAD and SSD, the larger the similarity value, the less similar it is, and the smaller the similarity value, the more similar it is. For NCC, the similarity value is a decimal between 0 and 1, and the larger the value (closer to 1), the more similar it is, and the smaller the value (closer to 0), the more dissimilar it is.

[0060] For example, taking SSD as an example, the similarity between two pixel blocks is shown in Formula 1:

[0061]

[0062] Wherein, x is the coordinate in the i-1th frame image; p is the optical flow (the initial value of the optical flow in the first iteration); W(x; p) represents the coordinate after the coordinate x is added to the optical flow p; I(W(x; p)) is the pixel value of the pixel block with coordinate W(x; p) in the i-1th frame image, T(x) is the pixel value of the pixel block with coordinate x in the i-1th frame image, and I(W(x; p))-T(x) is the difference between the above two pixel values.

[0063] Specifically, the similarity of brightness between pixel blocks may be calculated, or the similarity of pixel values ​​between pixel blocks may be calculated, which is not limited in the present application.

[0064] S102: Determine a second similarity between the first pixel block and the third pixel block in the i-th frame image.

[0065] The coordinates of the third pixel block are obtained from the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame. Each historical optical flow corresponds to a third pixel block, and the coordinates of the third pixel block (e.g., the coordinates of the pixel point in the upper left corner) are equal to the coordinates of the first pixel block (e.g., the coordinates of the pixel point in the upper left corner) plus the historical optical flow of the first pixel block, and the size of the third pixel block is the same as the size of the first pixel block. In other words, the historical optical flow of the first pixel block plus the coordinates of the first pixel block (e.g., the coordinates of the pixel point in the upper left corner) are used as the coordinates of the third pixel block in the i-th frame image (e.g., the coordinates of the pixel point in the upper left corner), and based on the coordinates, an area of ​​the same size as the first pixel block is selected in the i-th frame image as the third pixel block.

[0066] Since the movement of objects in the real world is consistent and coherent, in most cases, the current optical flow information can be inferred based on the historical optical flow information, or the historical optical flow information can be used to provide a reference for the calculation of the current optical flow information. This can also be called temporal consistency.

[0067] The historical optical flow of the first pixel block can be obtained when iteratively calculating the optical flow of the i-1th frame image, or it can be obtained by filtering and estimating the optical flow in the i-2th frame image and the optical flow in the i-1th frame image (for example, linear filtering). When these two historical optical flows are used at the same time, the third pixel block and the second similarity obtained are both two values.

[0068] For example, Figure 5 As shown, the coordinates of the first pixel block in the i-1th frame image are (1, 2), and the historical optical flow of the first pixel block is (2, 3). Then the coordinates of the third pixel block in the i-th frame image are (3, 5), and the size of the third pixel block is the same as that of the first pixel block, both of which are 3 pixels * 3 pixels.

[0069] The similarity between two pixel blocks is described in step S101 and will not be repeated here.

[0070] S103: Determine a third similarity between the first pixel block and the fourth pixel block in the i-th frame image.

[0071] The coordinates of the fourth pixel block are obtained from the optical flow of the adjacent pixel blocks of the first pixel block. Each current optical flow corresponds to a fourth pixel block, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the current optical flow, and the size of the fourth pixel block is the same as the size of the first pixel block. In other words, the current optical flow of the adjacent pixel blocks of the first pixel block plus the coordinates of the first pixel block (for example, the coordinates of the pixel point in the upper left corner) are used as the coordinates of the third pixel block in the i-th frame image (for example, the coordinates of the pixel point in the upper left corner), and based on the coordinates, an area of ​​the same size as the first pixel block is selected in the i-th frame image as the third pixel block.

[0072] Since rigid body motion is consistent, the current optical flow information of adjacent pixel blocks can be used as a reference to optimize the initial value of the iteration. This can also be called spatial consistency. The current optical flow refers to the optical flow obtained when iteratively calculating the optical flow of the i-th frame image.

[0073] As described above, when calculating the optical flow of the pixel block, the embodiment of the present application is performed in the order from left to right and from top to bottom. Therefore, for the first pixel block, the optical flow of the pixel blocks above and to the left of it is known. If the optical flow of the pixel block is calculated in the opposite order, the optical flow of the pixel blocks below and to the right of the first pixel block is known.

[0074] For example, Figure 6 As shown, each box represents a pixel block. For the sake of clarity, the overlapping parts between adjacent pixel blocks are not shown. The adjacent pixel blocks of the first pixel block include first-order adjacent pixel blocks and second-order adjacent pixel blocks. The first-order adjacent pixel blocks refer to the pixel blocks that are adjacent to the first pixel block and have obtained the optical flow, and the second-order adjacent pixel blocks refer to the pixel blocks that are separated from the first pixel block by one pixel block and have obtained the optical flow. Using the first-order adjacent pixel blocks to participate in the calculation can reduce the amount of calculation, and using the second-order adjacent pixel blocks to participate in the calculation can improve the accuracy.

[0075] For example, Figure 7As shown, the coordinates of the first pixel block in the i-1th frame image are (1, 2), the first pixel block has two first-order neighboring pixel blocks (coordinates are (1, 1) and (2, 1) respectively), and the current optical flow of the first-order neighboring pixel blocks is (2, 3), then there is a fourth pixel block in the i-th frame image, and the coordinates of the fourth pixel block are (3, 5), and the size of the fourth pixel block is the same as that of the first pixel block, both of which are 3 pixels * 3 pixels. For example, Figure 8 As shown, if the current optical flows of two first-order adjacent pixel blocks are (1, 4) and (2, 3) respectively, there are two fourth pixel blocks in the i-th frame image, and the coordinates of the fourth pixel blocks are (2, 6) and (3, 5) respectively.

[0076] The similarity between two pixel blocks is described in step S101 and will not be repeated here.

[0077] S104 . Obtain a target optical flow of the first pixel block from the (i-1)th frame image to the (i)th frame image according to at least one of the second similarity and the third similarity, the first similarity, and the gradient information of the first pixel block.

[0078] If the image is regarded as a two-dimensional discrete function, the gradient information is the derivative of this two-dimensional discrete function, which represents the distinction between the edge of the target and the background in the image.

[0079] The gradient information includes the sum of the gradient values ​​in the X direction, the sum of the squares of the gradient values ​​in the X direction, the sum of the gradient values ​​in the Y direction, the sum of the squares of the gradient values ​​in the Y direction, and the sum of the products of the gradient values ​​in the X direction and the Y direction.

[0080] The following describes how to obtain the gradient information of a pixel block.

[0081] First, convolve each pixel point of the first pixel block with the Sobel operator in the X direction to obtain the X direction gradient matrix, and convolve each pixel point of the first pixel block with the Sobel operator in the Y direction to obtain the Y direction gradient matrix. The size of the X direction gradient matrix and the Y direction gradient matrix is ​​the same as the size of the pixel block, that is, each pixel point corresponds to an X direction gradient value in the X direction gradient matrix and also corresponds to a Y direction gradient value in the Y direction gradient matrix.

[0082] For example, Fig. 9 As shown, the size of each pixel block A is 4 pixels * 4 pixels, and the size of the Sobel operator is 3 pixels * 3 pixels. Among them, C0 and C1 are preset convolution kernel coefficients. The 4*4=16 pixel points of the pixel block are traversed in sequence to calculate the convolution of each pixel point and the Sobel operator. If the Sobel operator is the X-direction Sobel operator, the X-direction gradient matrix is ​​obtained. If the Sobel operator is the X-direction Sobel operator, the X-direction gradient matrix is ​​obtained.

[0083] Then, all the gradient values ​​of the X-direction gradient matrix are cumulatively summed to obtain the sum of the X-direction gradient values, and each gradient value of the X-direction gradient matrix is ​​squared and cumulatively summed to obtain the sum of the squared gradient values ​​in the X-direction; all the gradient values ​​of the Y-direction gradient matrix are cumulatively summed to obtain the sum of the Y-direction gradient values, and each gradient value of the Y-direction gradient matrix is ​​squared and cumulatively summed to obtain the sum of the squared gradient values ​​in the Y-direction; the gradient values ​​at the same position of the X-direction gradient matrix and the Y-direction gradient matrix are multiplied and cumulatively summed to obtain the sum of the products of the X-direction and Y-direction gradient values. These cumulative summed values ​​are collectively referred to as the gradient information of the pixel block.

[0084] In a possible implementation, the highest similarity (i.e., the similarity with the lowest value) can be determined based on at least one of the second similarity and the third similarity and the first similarity, and the optical flow corresponding to the highest similarity is selected as the initial optical flow value. Combined with the gradient information of the first pixel block, the initial optical flow value is iteratively solved by approximate Gauss-Newton gradient descent, and the iterative result obtained when exiting because the exit condition is met is used as the target optical flow.

[0085] In another possible implementation, at least two highest similarities may be determined based on at least one of the second similarity and the third similarity and the first similarity, and the optical flows corresponding to the at least two highest similarities are selected as at least two initial optical flow values. In combination with the gradient information of the first pixel block, the at least two initial optical flow values ​​are respectively solved by approximate Gauss-Newton gradient descent iteration, and the exit condition is set to be the same. When exiting due to satisfying the exit condition, the iteration result with the minimum energy function is selected as the target optical flow.

[0086] The above exit conditions are set in the same implementation mode. Of course, other methods for determining the initial value of the optical flow can also be used, and this application does not limit this:

[0087] For example, a highest similarity may be determined from the second similarities, and another highest similarity may be determined from the third similarities, and the optical flows corresponding to the two highest similarities and the optical flow corresponding to the first similarity are used as three optical flow initial values.

[0088] A highest similarity may be determined from the second similarity and the first similarity, and then a highest similarity may be determined from the third similarity, and the optical flow corresponding to the highest similarity is used as the two optical flow initial values.

[0089] A highest similarity can be determined from the second similarity, and a highest similarity can be determined from the third similarity and the first similarity, and the optical flow corresponding to the highest similarity is used as the two optical flow initial values, and so on.

[0090] In the above process of determining the highest similarity, a threshold may also be combined to retain similarities less than the threshold.

[0091] The present application does not limit the number of highest similarities. For example, the optical flows corresponding to multiple better similarities can be selected as the initial optical flow value.

[0092] The following describes a process of performing an approximate Gauss-Newton gradient descent iterative solution on the initial value of the optical flow in combination with the gradient information of the first pixel block.

[0093] like Fig.10 As shown, the initial value of the optical flow and the gradient information of the first pixel block can be substituted into Formula 2 (or Formula 3) to perform an approximate Gauss-Newton gradient descent iterative solution. When the exit condition is not met, the result Δp calculated by Formula 2 (or Formula 3) is used as the negative feedback of the optical flow F (i.e., F=F-Δp), and is re-substituted into Formula 2 (or Formula 3) to perform an approximate Gauss-Newton gradient descent iterative solution until the exit condition is met. The optical flow entered into Formula 2 (or Formula 3) at the time of exit is the iterative result.

[0094]

[0095] Wherein, H is the Hessian matrix, which is obtained from the gradient information of the first pixel block. The size of the Hessian matrix is ​​2*2, H(0,0)=sum of squares of gradient values ​​in the X direction, H(0,1)=H(1,0)=sum of products of gradient values ​​in the X direction and Y direction, and H(1,1)=sum of squares of gradient values ​​in the Y direction; is the first gradient vector of the first pixel block in the i-1th frame image, including the gradient value in the X direction and the gradient value in the Y direction, that is, (gradient value in the X direction, gradient value in the Y direction); is the Jacobian matrix; x is the coordinate in the i-1th frame image; p is the optical flow (the initial value of the optical flow in the first iteration); W(x; p) represents the coordinate after the coordinate x is added to the optical flow p; I(W(x; p)) is the pixel value of the pixel block at the coordinate W(x; p) in the i-1th frame image, T(x) is the pixel value of the pixel block at the coordinate x in the i-1th frame image, and I(W(x; p))-T(x) is the difference between the above two pixel values.

[0096] Consider the equation 2 The part is normalized to improve the anti-interference ability of the optical flow acquisition method for the brightness change in the area, and formula 3 is obtained.

[0097]

[0098] Wherein, n is the number of pixels in the first pixel block, B is the second gradient vector of the first pixel block in the i-1th frame image, including the sum of the gradient values ​​in the X direction and the sum of the gradient values ​​in the Y direction, that is, (the sum of the gradient values ​​in the X direction, the sum of the gradient values ​​in the Y direction).

[0099] Accordingly, Formula 1 can be normalized to improve the anti-interference ability of the optical flow acquisition method for brightness changes in the area, and Formula 4 is obtained.

[0100]

[0101] Wherein, n is the number of pixels in the first pixel block.

[0102] Meeting the exit conditions includes but is not limited to reaching a preset number of iterations, the iteration result not meeting the convergence condition, the iteration result exceeding a preset threshold, etc.

[0103] Due to the limitation of the principle of gradient-based optical flow algorithm, it is easy to cause large errors in the gradient flat area. The present application also adds a confidence mechanism, the confidence of the optical flow with large error is low, and the confidence of the optical flow with small error is high.

[0104] Specifically, Fig.11 As shown, the optical flow acquisition method also includes:

[0105] S1101 . Determine a fourth similarity between a first pixel block and a fifth pixel block in an i-th frame image.

[0106] The coordinates of the fifth pixel block are obtained from the target optical flow. Specifically, the coordinates of the fifth pixel block are equal to the coordinates of the first pixel block plus the target optical flow, and the size of the fifth pixel block is the same as that of the first pixel block.

[0107] In this step, a pixel block in the i-th frame image is obtained according to the optical flow and the first pixel block, and the similarity between two pixel blocks is calculated, which is similar to step S102 and step S103 and will not be repeated here.

[0108] S1102: Determine the confidence of the target optical flow according to the first similarity and the fourth similarity.

[0109] The first similarity is equivalent to the similarity between the same position (first pixel block) in the i-th frame image and the i+1-th frame image before iterative calculation, and the fourth similarity is equivalent to the similarity between the same target in the i-th frame image and the i+1-th frame image after iterative calculation.

[0110] Specifically, Fig.12 As shown, step S1102 includes:

[0111] S11021: If the ratio of the value of the fourth similarity to the value of the first similarity is greater than the first threshold, it is determined that the target optical flow is of low confidence. Otherwise, execute step S11022.

[0112] The ratio of the fourth similarity value to the first similarity value is an evaluation of relative accuracy. As mentioned above, the smaller the similarity value, the higher the similarity. Therefore, the smaller the ratio of the fourth similarity value to the first similarity value, the more accurate the tracking of the same target is after iteration, otherwise it is less accurate.

[0113] S11022: If the value of the fourth similarity is greater than the second threshold, determine that the target optical flow is of low confidence. Otherwise, execute step S11023.

[0114] Comparing the value of the fourth similarity with the threshold is an evaluation of absolute accuracy. The smaller the value of the fourth similarity is, the more accurate the tracking of the same target is after iteration. Otherwise, it is less accurate.

[0115] S11023. Determine that the target optical flow is of high confidence.

[0116] The optical flow and its confidence obtained by the above optical flow acquisition method are sparse, which can already meet the application requirements of motion detection and motion tracking. For example, based on the above information, the motion of the subject in the field of view can be detected, and the direction and speed of the subject's motion can be obtained, thereby realizing the motion tracking function.

[0117] The optical flow obtained by the above optical flow acquisition method can also be densified to obtain pixel-level optical flow, so as to achieve registration and alignment between pixels, which is used to assist in improving the performance of multi-frame denoising, multi-frame super-resolution, multi-frame exposure fusion, and multi-frame anti-mosaic.

[0118] The optical flow algorithm can calculate the motion relationship between the pixel blocks of two frames of images, including both the global motion of the entire field of view and the local motion of objects in the scene. Therefore, the optical flow information can be considered as a one-to-one mapping relationship between the pixel blocks of the two frames of images, that is, the registration and alignment between the pixel blocks of the images.

[0119] For applications such as multi-frame denoising, multi-frame super-resolution, multi-frame exposure fusion, and multi-frame anti-mosaic in image processing, the optical flow algorithm can provide the registration and alignment function between pixel blocks of the image, assisting in improving the effect and reducing the amount of matching calculations. If the registration and alignment between pixels can be achieved, the performance will be further improved.

[0120] like Fig.13 As shown, the above optical flow acquisition method also includes:

[0121] S1301 . Determine a weighting coefficient of a first pixel block according to a first pixel in an (i-1)th frame and a target optical flow of a first pixel block including the first pixel.

[0122] Assuming that the coordinates of the first pixel in the i-1th frame are (x0, y0), the pixel value is P0, and the optical flow of the first pixel block is (dx, dy), then the coordinates of the first pixel are added to the optical flow of the first pixel block to obtain the second pixel in the i-th frame image. The coordinates of the second pixel are (x0+dx, y0+dy), and the pixel value of the second pixel is P1.

[0123] Determine a weighting coefficient of the first pixel block according to the pixel value of the first pixel and the pixel value of the second pixel Wherein, j represents the jth first pixel block. When multiple first pixel blocks contain the first pixel, the value range of j is greater than 1; abs represents the absolute value.

[0124] S1302: Determine an optical flow of a first pixel according to the weighting coefficient and a target optical flow of the first pixel block.

[0125] Optical flow of the first pixel Among them, Flow j represents the j-th first pixel block.

[0126] It should be noted that when the above optical flow acquisition method is applied to multi-frame denoising, a certain pre-denoising process can be performed to improve the interference of noise on the optical flow calculation. When the above optical flow acquisition method is applied to multi-frame exposure fusion, the images with different exposures can be normalized to meet the brightness consistency assumption required by the optical flow algorithm.

[0127] In addition, due to the limitation of processor computing power, there is a gap between the image rendering rate and the display frame rate. The time domain super-resolution interpolation function can be realized with the help of the results of the optical flow acquisition method of the present application.

[0128] like Fig.14 As shown, the above optical flow acquisition method also includes:

[0129] S1401. Insert an intermediate image between the i-1th frame image and the i-th frame image according to the target optical flow.

[0130] The optical flow algorithm obtains the motion change relationship between two real frames of images, and on this basis, it can infer the motion relationship at a certain moment (or multiple moments) between the moments corresponding to the two frames of images. That is, the super-resolution interpolation result can be obtained at a low cost through the reference frame image (such as an earlier image) and the mapping relationship inferred by the optical flow.

[0131] Assume that there is a first image I1 and a second image I2, and we want to obtain an intermediate image I1.5 between the first image I1 and the second image I2. Fig.13 As described above, the optical flow of the pixel point can be obtained according to the target optical flow of the pixel block. For example, the optical flow of a pixel point A in the first image I1 is F1_2.

[0132] Since the time between the two frames is short enough, the speed of the target in the image can be approximately considered to be linear, so the optical flow F1_2 can be halved to obtain the optical flow F1_1.5 = (F1_2) / 2 between the pixel A from the first image I1 to the intermediate image I1.5. The optical flow F1_1.5 represents the transformation relationship between the pixel A from the first image I1 to the intermediate image I1.5. The intermediate image I1.5 can be obtained by performing the above processing on all the pixels in the first image I1.

[0133] It should be noted that for occlusion scenes and physical interaction scenes, some special considerations need to be taken into account when inferring the optical flow information between two frames of images. For example, occlusion detection can be performed through a bidirectional optical flow acquisition method. Specifically, the optical flow of the first pixel block from the i-1th frame image to the i-th frame image can be solved first, denoted as F; then the optical flow of the first pixel block from the i-th frame image to the i-1th frame image can be solved, denoted as F'. Theoretically, if there is no occlusion and the optical flow calculation is relatively accurate, the target at a certain position P on the i-1th frame image can be mapped to the position P' on the i-th frame image by the optical flow F; and then mapped back to the position P" on the i-1th frame image by the optical flow F', and P" basically coincides with P. Otherwise, it can be inferred that occlusion has occurred. In occlusion scenes, optical flows with higher confidence are usually used.

[0134] The embodiment of the present application also provides an optical flow acquisition device, which is used to implement the above-mentioned various methods. The optical flow acquisition device can be a mobile phone, a tablet, a drone, a car, an electric car, etc.

[0135] It is understandable that, in order to realize the above functions, the optical flow acquisition device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0136] The embodiment of the present application can divide the optical flow acquisition device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0137] Fig.15 1 shows a schematic diagram of the structure of an optical flow acquisition device 150. The optical flow acquisition device 150 includes a determination module 1501, an acquisition module 1502 and an insertion module 1503. The determination module 1501 is used to implement the above method embodiment. Figure 1 Steps S101-S103 in Fig.11 Steps S101-S103, S1101-S1102 in Fig.12 Steps S101-S103, S1101, S11021-S11023 in Fig.13 Steps S101-S103, S1301-S1302 in Fig.14 The acquisition module 1502 is used to implement the above method embodiment. Figure 1 In step S104, Fig.11 In step S104, Fig.13 The frame insertion module 1503 is used to implement the above method embodiment. Fig.14 Step S1401 in .

[0138] Exemplarily, the determination module 1501 is used to determine a first similarity between a first pixel block in the (i-1)th frame image and a second pixel block in the (i)th frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer.

[0139] The determination module 1501 is also used to perform at least one of the following two processes: determining a second similarity between the first pixel block and a third pixel block in the i-th frame image, wherein the coordinates of the third pixel block are obtained by a historical optical flow of the first pixel block from the i-2-th frame image to the i-1-th frame; and determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, wherein the coordinates of the fourth pixel block are obtained by an optical flow of adjacent pixel blocks of the first pixel block.

[0140] The acquisition module 1502 is used to obtain the target optical flow of the first pixel block from the i-1th frame image to the i-th frame image according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block.

[0141] In a possible implementation, the coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame.

[0142] In a possible implementation, the coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of the adjacent pixel blocks of the first pixel block.

[0143] In a possible implementation, the acquisition module 1502 is specifically used to: determine the highest similarity based on at least one of the second similarity and the third similarity and the first similarity, and select the optical flow corresponding to the highest similarity as the initial optical flow value; combine the gradient information of the first pixel block, perform an approximate Gauss-Newton gradient descent iterative solution on the initial optical flow value, and use the iterative result obtained when exiting because the exit condition is met as the target optical flow.

[0144] In a possible implementation, the acquisition module 1502 is specifically used to: determine at least two highest similarities based on at least one of the second similarity and the third similarity and the first similarity, and select optical flows corresponding to the at least two highest similarities as at least two optical flow initial values; in combination with the gradient information of the first pixel block, perform approximate Gauss-Newton gradient descent iterative solutions on the at least two optical flow initial values, and set the exit condition to be the same. When exiting due to satisfying the exit condition, select the iterative result with the smallest energy function as the target optical flow.

[0145] In a possible implementation, the gradient information includes the sum of the X-direction gradient values, the sum of the squares of the X-direction gradient values, the sum of the Y-direction gradient values, the sum of the squares of the Y-direction gradient values, and the sum of the products of the X-direction and Y-direction gradient values. The acquisition module 1502 is further used to: perform convolution on each pixel point of the first pixel block and the X-direction Sobel operator to obtain an X-direction gradient matrix, perform convolution on each pixel point of the first pixel block and the Y-direction Sobel operator to obtain a Y-direction gradient matrix; perform cumulative summation on all gradient values ​​of the X-direction gradient matrix to obtain the sum of the X-direction gradient values, perform cumulative summation on each gradient value of the X-direction gradient matrix after squaring, obtain the sum of the squares of the X-direction gradient values; perform cumulative summation on all gradient values ​​of the Y-direction gradient matrix to obtain the sum of the Y-direction gradient values, perform cumulative summation on each gradient value of the Y-direction gradient matrix after squaring, obtain the sum of the squares of the Y-direction gradient values; perform cumulative summation on the gradient values ​​at the same position of the X-direction gradient matrix and the Y-direction gradient matrix to obtain the sum of the products of the X-direction and Y-direction gradient values.

[0146] In a possible implementation, the determination module 1501 is further used to: determine a fourth similarity between the first pixel block and the fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained by the target optical flow; and determine the confidence of the target optical flow based on the first similarity and the fourth similarity.

[0147] In a possible implementation, the determination module 1501 is specifically used to: if the ratio of the value of the fourth similarity to the value of the first similarity is greater than a first threshold, determine that the target optical flow is low confidence; otherwise, if the value of the fourth similarity is greater than a second threshold, determine that the target optical flow is low confidence; otherwise, determine that the target optical flow is high confidence.

[0148] In a possible implementation, the frame insertion module 1503 is configured to insert an intermediate image between the (i-1)th frame image and the (i)th frame image according to the target optical flow.

[0149] In this embodiment, the optical flow acquisition device 150 is presented in the form of dividing various functional modules in an integrated manner. The "module" here may refer to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a circuit, a processor and a memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0150] Since the optical flow acquisition device 150 provided in this embodiment can execute the above method, the technical effects that can be obtained can refer to the above method embodiments, which will not be repeated here.

[0151] like Fig.16 As shown, the embodiment of the present application further provides an optical flow acquisition device, the optical flow acquisition device 160 includes a processor 1602 and a memory 1601, the processor 1602 and the memory 1601 are coupled via a bus 1603, the memory 1601 stores computer instructions, when the processor 1602 executes the computer instructions in the memory 1601, the execution Figure 1 , Figure 11-Figure 14 The optical flow acquisition method in .

[0152] The present application also provides a chip, including: a processor and an interface, for calling and running a computer program stored in the memory from the memory, and executing Figure 1 , Figure 11-Figure 14 The optical flow acquisition method in .

[0153] The present application also provides a computer-readable storage medium in which instructions are stored. When the instructions in the computer-readable storage medium are executed on a computer or a processor, the computer or the processor executes Figure 1 , Figure 11-Figure 14 The optical flow acquisition method in .

[0154] The present application also provides a computer program product including instructions, which, when executed on a computer or processor, causes the computer or processor to execute Figure 1 , Figure 11-Figure 14 The optical flow acquisition method in .

[0155] The present application embodiment provides a chip system, which includes a processor for executing an optical flow acquisition device. Figure 1 , Figure 11-Figure 14 The optical flow acquisition method in .

[0156] In one possible design, the chip system also includes a memory, which is used to store necessary program instructions and data. The chip system may include a chip, an integrated circuit, or a chip and other discrete devices, which is not specifically limited in the embodiments of the present application.

[0157] Among them, the optical flow acquisition device, chip, computer storage medium, computer program product or chip system provided in the present application is used to execute the method described above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the implementation methods provided above and will not be repeated here.

[0158] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0159] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0160] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0161] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for obtaining optical flow, characterized in that: include: Determine a first similarity between a first pixel block in an i-1th frame image and a second pixel block in an i-th frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer; Perform at least one of the following two procedures: Determine a second similarity between the first pixel block and a third pixel block in the i-th frame image, wherein the coordinates of the third pixel block are obtained by a historical optical flow of the first pixel block from the i-2-th frame image to the i-1-th frame; Determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, wherein the coordinates of the fourth pixel block are obtained by optical flows of adjacent pixel blocks of the first pixel block; A target optical flow of the first pixel block from the (i-1)th frame image to the (i)th frame image is obtained according to at least one of the second similarity and the third similarity, the first similarity, and the gradient information of the first pixel block.

2. The method according to claim 1, characterized in that The coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame.

3. The method according to any one of claims 1 to 2, characterized in that: The coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of the adjacent pixel blocks of the first pixel block.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining, according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block, a target optical flow of the first pixel block from the (i-1)th frame image to the i-th frame image comprises: Determine a highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and select an optical flow corresponding to the highest similarity as an initial optical flow value; Combined with the gradient information of the first pixel block, the initial value of the optical flow is iteratively solved by approximate Gauss-Newton gradient descent, and the iterative result obtained when exiting because the exit condition is met is used as the target optical flow.

5. The method according to any one of claims 1 to 3, characterized in that: The obtaining, according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block, a target optical flow of the first pixel block from the (i-1)th frame image to the i-th frame image comprises: Determine at least two highest similarities according to the first similarity and at least one of the second similarity and the third similarity, and select optical flows corresponding to the at least two highest similarities respectively as at least two optical flow initial values; Combined with the gradient information of the first pixel block, the at least two initial optical flow values ​​are respectively solved by approximate Gauss-Newton gradient descent iteration, and the exit conditions are set to be the same. When exiting due to meeting the exit conditions, the iterative result with the smallest energy function is selected as the target optical flow.

6. The method according to any one of claims 1 to 5, characterized in that: The gradient information includes the sum of the gradient values ​​in the X direction, the sum of the squares of the gradient values ​​in the X direction, the sum of the gradient values ​​in the Y direction, the sum of the squares of the gradient values ​​in the Y direction, and the sum of the products of the gradient values ​​in the X direction and the Y direction. The method further includes: Convolve each pixel point of the first pixel block with the Sobel operator in the X direction to obtain an X-direction gradient matrix, and convolve each pixel point of the first pixel block with the Sobel operator in the Y direction to obtain a Y-direction gradient matrix; Cumulatively summing all the gradient values ​​of the X-direction gradient matrix to obtain the sum of the X-direction gradient values, and squaring each gradient value of the X-direction gradient matrix and then cumulatively summing them to obtain the sum of the squares of the X-direction gradient values; All gradient values ​​of the Y-direction gradient matrix are cumulatively summed to obtain the sum of the Y-direction gradient values, and each gradient value of the Y-direction gradient matrix is ​​squared and then cumulatively summed to obtain the sum of the squares of the Y-direction gradient values; The gradient values ​​at the same position of the X-direction gradient matrix and the Y-direction gradient matrix are multiplied and then accumulated to obtain the product sum of the X-direction and Y-direction gradient values.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determining a fourth similarity between the first pixel block and a fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained by the target optical flow; The confidence of the target optical flow is determined according to the first similarity and the fourth similarity.

8. The method according to claim 7, characterized in that Determining the confidence of the target optical flow according to the first similarity and the fourth similarity includes: If the ratio of the value of the fourth similarity to the value of the first similarity is greater than a first threshold, determining that the target optical flow is of low confidence; Otherwise, if the value of the fourth similarity is greater than the second threshold, determining that the target optical flow is of low confidence; Otherwise, the target optical flow is determined to be high confidence.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: An intermediate image is inserted between the i-1th frame image and the i-th frame image according to the target optical flow.

10. An optical flow acquisition device, characterized in that: include: A determination module, used to determine a first similarity between a first pixel block in an i-1th frame image and a second pixel block in an i-th frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block, and i is a positive integer; The determining module is further configured to perform at least one of the following two processes: Determine a second similarity between the first pixel block and a third pixel block in the i-th frame image, wherein the coordinates of the third pixel block are obtained by a historical optical flow of the first pixel block from the i-2-th frame image to the i-1-th frame; Determining a third similarity between the first pixel block and a fourth pixel block in the i-th frame image, wherein the coordinates of the fourth pixel block are obtained by optical flows of adjacent pixel blocks of the first pixel block; An acquisition module is used to obtain a target optical flow of the first pixel block from the (i-1)th frame image to the (i)th frame image according to at least one of the second similarity and the third similarity, the first similarity and the gradient information of the first pixel block.

11. The optical flow acquisition device according to claim 10, characterized in that: The coordinates of the third pixel block are equal to the coordinates of the first pixel block plus the historical optical flow of the first pixel block from the i-2th frame image to the i-1th frame.

12. The optical flow acquisition device according to any one of claims 10 to 11, characterized in that: The coordinates of the fourth pixel block are equal to the coordinates of the first pixel block plus the optical flow of the adjacent pixel blocks of the first pixel block.

13. The optical flow acquisition device according to any one of claims 10 to 12, characterized in that: The acquisition module is specifically used for: Determine a highest similarity according to at least one of the second similarity and the third similarity and the first similarity, and select an optical flow corresponding to the highest similarity as an initial optical flow value; Combined with the gradient information of the first pixel block, the initial value of the optical flow is iteratively solved by approximate Gauss-Newton gradient descent, and the iterative result obtained when exiting because the exit condition is met is used as the target optical flow.

14. The optical flow acquisition device according to any one of claims 10 to 12, characterized in that: The acquisition module is specifically used for: Determine at least two highest similarities according to the first similarity and at least one of the second similarity and the third similarity, and select optical flows corresponding to the at least two highest similarities respectively as at least two optical flow initial values; Combined with the gradient information of the first pixel block, the at least two initial optical flow values ​​are respectively solved by approximate Gauss-Newton gradient descent iteration, and the exit conditions are set to be the same. When exiting due to meeting the exit conditions, the iterative result with the smallest energy function is selected as the target optical flow.

15. The optical flow acquisition device according to any one of claims 10 to 14, characterized in that: The gradient information includes the sum of the gradient values ​​in the X direction, the sum of the squares of the gradient values ​​in the X direction, the sum of the gradient values ​​in the Y direction, the sum of the squares of the gradient values ​​in the Y direction, and the sum of the products of the gradient values ​​in the X direction and the Y direction. The acquisition module is further used for: Convolve each pixel point of the first pixel block with the Sobel operator in the X direction to obtain an X-direction gradient matrix, and convolve each pixel point of the first pixel block with the Sobel operator in the Y direction to obtain a Y-direction gradient matrix; Cumulatively summing all the gradient values ​​of the X-direction gradient matrix to obtain the sum of the X-direction gradient values, and squaring each gradient value of the X-direction gradient matrix and then cumulatively summing them to obtain the sum of the squares of the X-direction gradient values; All gradient values ​​of the Y-direction gradient matrix are cumulatively summed to obtain the sum of the Y-direction gradient values, and each gradient value of the Y-direction gradient matrix is ​​squared and then cumulatively summed to obtain the sum of the squares of the Y-direction gradient values; The gradient values ​​at the same position of the X-direction gradient matrix and the Y-direction gradient matrix are multiplied and then accumulated to obtain the product sum of the X-direction and Y-direction gradient values.

16. The optical flow acquisition device according to any one of claims 10 to 15, characterized in that: The determining module is also used for: Determining a fourth similarity between the first pixel block and a fifth pixel block in the i-th frame image, wherein the coordinates of the fifth pixel block are obtained by the target optical flow; The confidence of the target optical flow is determined according to the first similarity and the fourth similarity.

17. The optical flow acquisition device according to claim 16, characterized in that: The determination module is specifically used for: If the ratio of the value of the fourth similarity to the value of the first similarity is greater than a first threshold, determining that the target optical flow is of low confidence; Otherwise, if the value of the fourth similarity is greater than the second threshold, determining that the target optical flow is of low confidence; Otherwise, the target optical flow is determined to be high confidence.

18. The optical flow acquisition device according to any one of claims 10 to 17, characterized in that: The optical flow acquisition device further includes a frame insertion module, which is used to: An intermediate image is inserted between the i-1th frame image and the i-th frame image according to the target optical flow.

Images