Image optical flow determination method, system, device and storage medium

Abstract

The embodiments of the present application disclose a method, system, device and storage medium for determining the optical flow of an image. The technical solution provided by the embodiments of the present application obtains a first optical flow map, determines the second pixel coordinates corresponding to the first pixel coordinates of the first image in the second image according to the first optical flow map; determines the four vertex integer pixel coordinates closest to the second pixel coordinate, and calculates the distance weights between the second pixel coordinate and the integer pixel coordinates of each vertex; determines the weighted optical flow information of the integer pixel coordinates of each vertex corresponding to the second pixel coordinate in the first image according to the distance weight and the inverse value of the first pixel coordinate, and calculates the optical flow of the second optical flow map based on the weighted optical flow information. By adopting the above technical means, the optical flow map in one direction between two images can be quickly calculated based on the optical flow map in another direction, thereby improving the efficiency of optical flow calculation and thus improving the efficiency of image processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, system, device, and storage medium for determining image optical flow. Background Art

[0002] Optical flow describes the motion between two image frames and is often used for image enhancements such as warping, interpolation, and super-resolution. For two images A and B with the same resolution and timestamps t and t+1, respectively, the motion from A to B is generally referred to as forward optical flow, and the motion from B to A is referred to as backward optical flow. Forward optical flow stores the positional offset of each pixel in A at time t relative to the position of each pixel in B at time t+1, while backward optical flow stores the positional offset of each pixel in B at time t+1 relative to the position of each pixel in A at time t. In some scenarios, not only forward or backward optical flow is required, but also both. For example, when interpolating between two video frames A and B, two interpolation results are calculated based on the forward optical flow and A, and the backward optical flow and B, respectively. These two interpolation results are then fused to achieve a better interpolation effect. Another scenario is that, given an image A at timestamp t and a reverse optical flow F, when the image B at time t+1 is unknown, A is warped according to the reverse optical flow F to obtain the deformation result of A to B. This operation is achieved by retrieving the position of the pixel corresponding to each coordinate of B at time t+1 in the reverse optical flow F, sampling A, and generating a complete image B. On this basis, if you want to calculate the position of the key points of A (such as the key points of the face) after the optical flow displacement, you need to first invert the reverse optical flow to obtain the forward optical flow G, and retrieve the motion information in the forward optical flow G according to the position of the key points at time t to calculate its position at time t+1.

[0003] By calculating the forward optical flow and reverse optical flow of the two images respectively, image processing services such as image interpolation and deformation can be performed based on the forward optical flow and reverse optical flow.

[0004] However, the related optical flow determination scheme needs to calculate the forward optical flow and the reverse optical flow separately, which has a relatively high computational complexity and a large time overhead of the entire process, affecting the image processing efficiency. Summary of the Invention

[0005] The embodiments of the present application provide a method, system, device and storage medium for determining the optical flow of an image, which can quickly calculate the optical flow map in one direction between two images based on the optical flow map in another direction, simplify the optical flow calculation process, and solve the technical problem that the optical flow determination process is cumbersome and time-consuming.

[0006] In a first aspect, an embodiment of the present application provides a method for determining image optical flow, comprising:

[0007] Obtaining a first optical flow map, and determining, based on the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image, wherein the first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image;

[0008] Determine integer pixel coordinates adjacent to the second pixel coordinate, and calculate distance weights between the second pixel coordinate and each integer pixel coordinate;

[0009] Based on the distance weight and the inverse value of the first pixel coordinate, weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image is determined, and the optical flow of a second optical flow map is calculated based on the weighted optical flow information. The second optical flow map is used to store position offset information of each pixel coordinate in the second image in the first image.

[0010] In a second aspect, an embodiment of the present application provides an image optical flow determination system, comprising:

[0011] a coordinate determination module configured to obtain a first optical flow map and determine, based on the first optical flow map, a second pixel coordinate corresponding to a first pixel coordinate of the first image in the second image, wherein the first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image;

[0012] a weight calculation module configured to determine integer pixel coordinates adjacent to the second pixel coordinate and calculate distance weights between the second pixel coordinate and each integer pixel coordinate;

[0013] The optical flow calculation module is configured to determine the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image based on the distance weight and the inverse value of the first pixel coordinate, and calculate the optical flow of a second optical flow map based on the weighted optical flow information. The second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image.

[0014] In a third aspect, an embodiment of the present application provides an image optical flow determination device, comprising:

[0015] memory and one or more processors;

[0016] The memory is configured to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image optical flow determination method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they are configured to execute the image optical flow determination method as described in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the image optical flow determination method as described in the first aspect.

[0020] The embodiment of the present application obtains a first optical flow map, determines the corresponding second pixel coordinates of a first pixel coordinate of the first image in the second image based on the first optical flow map, and the first optical flow map is used to store the position offset information of each pixel coordinate in the first image in the second image; determines the integer pixel coordinates adjacent to the second pixel coordinate, and calculates the distance weights between the second pixel coordinate and each integer pixel coordinate; determines the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image based on the distance weights and the inverse value of the first pixel coordinate, and calculates the optical flow of a second optical flow map based on the weighted optical flow information, and the second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image. By using the above technical means, the second pixel coordinates corresponding to the first pixel coordinate of the first image in the second image are determined based on the first optical flow map, the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate is determined in the first image, and the optical flow of the second optical flow map is calculated based on the weighted optical flow information. In this way, the optical flow map in one direction between the two images can be quickly calculated based on the optical flow map in the other direction, simplifying the optical flow calculation process, avoiding the situation where the calculation complexity of the two optical flow maps is high, improving the efficiency of optical flow calculation, and thus improving the efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of a method for determining image optical flow provided by an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of image A in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of image B in an embodiment of the present application;

[0024] Figure 4 This is the image A restored based on the forward optical flow in the embodiment of the present application. ′ Schematic diagram;

[0025] Figure 5 Schematic diagram of reverse optical flow from image B to image A in an embodiment of the present application;

[0026] Figure 6 is a flowchart for determining the second pixel coordinate in an embodiment of the present application;

[0027] Figure 7 Schematic diagram of the forward optical flow from image A to image B in an embodiment of the present application;

[0028] Figure 8 This is a schematic diagram of the structure of an image optical flow determination system provided in an embodiment of the present application;

[0029] Figure 9 It is a structural diagram of an image optical flow determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should also be noted that, for ease of description, only parts related to the present application, not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0031] The image optical flow determination method provided in this application aims to determine the second pixel coordinates corresponding to the first pixel coordinates of the first image in the second image through a first optical flow map, determine the weighted optical flow information of the integer pixel coordinates of each vertex corresponding to the second pixel coordinates in the first image, and then calculate the optical flow of the second optical flow map based on the weighted optical flow information. In this way, the optical flow map in one direction between the two images can be quickly calculated based on the optical flow map in another direction, thereby simplifying the optical flow calculation process.

[0032] The concept of optical flow frequently appears in image processing algorithms. It describes the motion between two frames and is commonly used for image enhancements such as deformation, interpolation, and super-resolution. For two images A and B with the same resolution and timestamps t and t+1, respectively, the motion from A to B is generally referred to as forward optical flow, and the motion from B to A is referred to as backward optical flow. Forward optical flow stores the positional offset of each pixel in A in B, while backward optical flow stores the positional offset of each pixel in B in A.

[0033] Although the two optical flows are defined as inverses of each other, since optical flow values ​​can be arbitrary floating-point numbers and must be stored in integer coordinates within the image, simply negating the optical flow value in one direction and storing it in the reverse optical flow map may result in non-integer coordinates. Optical flow is typically stored in integer pixel coordinates, making the two inverses inversely proportional. Forward optical flow stores motion information based on the pixel position of A as the reference coordinate, and this motion information must be retrieved using the coordinates at time t. Reverse optical flow stores motion information based on the pixel position of B as the reference coordinate, and this motion information must be retrieved using the coordinates at time t+1. In other words, given the coordinates at time t, to determine the pixel position at time t+1, forward optical flow is required. Given the coordinates at time t+1, to determine the pixel position at time t, reverse optical flow is required.

[0034] In some scenarios, not only forward or reverse optical flow is required, but also two directions of optical flow simultaneously. In one scenario, when interpolating between two video frames A and B, two interpolation results are calculated based on the forward optical flow and A, and the reverse optical flow and B, respectively. These two interpolation results are then fused to achieve a better interpolation effect. Another scenario is given an image A at timestamp t and a reverse optical flow F, and when image B at time t+1 is unknown, A is warped using the reverse optical flow F to obtain a deformation of A toward B. This operation is achieved by retrieving the corresponding pixel position in A for each coordinate in the reverse optical flow based on the coordinates at time t+1, sampling A, and generating the complete image B. Based on this, if one wishes to calculate the position of key points in A (e.g., facial key points) after optical flow displacement, one must first invert the reverse optical flow to obtain the forward optical flow G. Based on the key point position at time t, motion information is retrieved from the forward optical flow to calculate its position at time t+1.

[0035] When forward and reverse optical flows are needed at the same time, some algorithms will calculate the optical flows twice separately. However, this will increase the time overhead. The related methods of estimating reverse optical flows from forward optical flows often have too high computational complexity and are not suitable for mobile terminal processing. Based on this, an image optical flow determination method is provided in an embodiment of the present application. The method can quickly calculate the optical flow map in one direction between two images based on the optical flow map in another direction, and realize real-time operation of optical flow calculation on the mobile terminal. It is suitable for scenarios where image processing algorithms that require both forward and reverse optical flows are deployed on the mobile terminal, and solves the technical problem of the cumbersome and time-consuming optical flow determination process.

[0036] Example:

[0037] Figure 1A flowchart of a method for determining image optical flow provided in an embodiment of the present application is provided. The method provided in this embodiment can be performed by an image optical flow determination device. The image optical flow determination device can be implemented through software and / or hardware. The image optical flow determination device can be composed of two or more physical entities, or a single physical entity. Generally speaking, the image optical flow determination device can be a mobile device such as a computer, mobile phone, or tablet.

[0038] The following description is made by taking the image optical flow determination device as an example of the main body of the image optical flow determination method. Figure 1 , the image optical flow determination method specifically includes:

[0039] S110: Obtain a first optical flow map, and determine, based on the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image. The first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image.

[0040] When calculating the optical flow of an image, the present application obtains an optical flow map in the opposite direction according to the corresponding reverse optical flow that needs to be calculated. The obtained optical flow map is defined as the first optical flow map, and the optical flow map in the opposite direction to be calculated is defined as the second optical flow map. It should be noted that the present application can solve the reverse optical flow based on the forward optical flow, and can also solve the forward optical flow based on the reverse optical flow. The calculation method is exactly the same. If the input is the forward optical flow, the output is the reverse optical flow. If the input is the reverse optical flow, the output is the forward optical flow.

[0041] Based on the acquired first optical flow map, the first optical flow map is used to store the position offset information of each pixel coordinate in the first image in the second image. Then, through the position offset information recorded by the first optical flow map and combined with the pixel coordinates of the first image (defined as the first pixel coordinates), the pixel coordinates corresponding to the first pixel coordinates of the first image in the second image can be determined, which are defined as the second pixel coordinates.

[0042] For two adjacent frames of image A (time stamp is t) and image B (time stamp is t+1), set the first image to B and the second image to A. Image A is as follows: Figure 2 As shown, image B is Figure 3 As shown. Define the reverse optical flow map F of A relative to B, and define the forward optical flow map G of B relative to A. Among them, B is obtained by deforming A according to the reverse optical flow map F. After obtaining the forward optical flow map G, image B can be deformed according to the optical flow map G to reconstruct image A to obtain image A'. The closer image A' is to A, the more accurate the forward optical flow map G is calculated. Among them, image A' is as follows Figure 4 As shown, the reverse optical flow graph F is as follows Figure 5 shown.

[0043] Based on image A, image B, and the reverse optical flow F from B to A, assuming that the width and height of the three are W and H, each pixel coordinate of the optical flow graph F stores two floating-point numbers, which record the horizontal and vertical offset values ​​of the coordinates of the pixel at each coordinate of image B corresponding to the pixel in image A. Based on this characteristic, the second pixel coordinate can be determined. Figure 6 , determining, according to the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image, comprising:

[0044] S1101, obtaining horizontal axis offset information and vertical axis offset information stored in a first pixel coordinate of a first image corresponding to a first optical flow map;

[0045] S1102 : Calculate the second pixel coordinate corresponding to the first pixel coordinate in the second image based on the first pixel coordinate, the horizontal axis offset information, and the vertical axis offset information.

[0046] Assume that the first optical flow map stores the horizontal axis offset information vx=F[i, j, 0] and the vertical axis offset information vy=F[i, j, 1] corresponding to the first pixel coordinate of the first image. Then for the pixel coordinate B of the first image i,j , and the pixel coordinates A of the second image i,j , the following equation needs to be satisfied:

[0047]

[0048] Among them B i,j and A i,j Here, i represents the horizontal axis coordinate, and j represents the vertical axis coordinate.

[0049] By scanning the first optical flow graph F, traversing columns [0, H-1] for i and rows [0, W-1] for j, we can determine the horizontal and vertical offset information vx = F[i, j, 0] and vy = F[i, j, 1] stored for each first pixel coordinate. vx and vy are both floating-point numbers representing the optical flow stored in F[i, j]. We can then obtain cx = j + vx; cy = i + vy, where cx and cy represent the corresponding coordinates of the pixel coordinate of B[i, j] in graph A, i.e., the second pixel coordinate corresponding to the first pixel coordinate in the second image.

[0050] S120 : Determine integer pixel coordinates adjacent to the second pixel coordinate, and calculate distance weights between the second pixel coordinate and each integer pixel coordinate.

[0051] Furthermore, based on the determined second pixel coordinate, it is necessary to determine the integer pixel coordinates adjacent to the second pixel coordinate, and by calculating the distance weights between the second pixel coordinate and each adjacent integer pixel coordinate, determine the optical flow assigned to each adjacent integer pixel coordinate corresponding to the second pixel coordinate in the first image.

[0052] It should be noted that in image processing and computer vision, the pixel coordinates of an image are accessed through integer indices. Since each pixel in the optical flow map F stores the coordinate offset (floating point number) from image B to image A, these offsets need to be converted back to integer coordinates in image A in order to correctly generate the corresponding optical flow map G.

[0053] Specifically, determining integer pixel coordinates adjacent to the second pixel coordinate includes:

[0054] The horizontal axis coordinate value and the vertical axis coordinate value of the second pixel coordinate are rounded down, and the integer pixel coordinates of the four vertices closest to the second pixel coordinate are determined as the integer pixel coordinates according to the rounding results.

[0055] For the above-mentioned second pixel coordinates (cx, cy), the present application calculates the distance weight by determining the integer pixel coordinates of the four nearest vertices. In order to determine the integer coordinates of the four nearest corners around it, by rounding down, we can get cx_0 = floor (cx), cy_0 = floor (cy). Further calculation cx_1 = cx_0 + 1, cy_1 = cy_0 + 1. Rounding down can determine the integer horizontal coordinate cx_0 to the left of cx, and the integer vertical coordinate cy_0 above cy (the value of cy_0 is smaller than cy, but in the image coordinate system, the smaller coordinate value in the y direction is higher, so in the image coordinate system, the integer coordinate above cy is cy_0). Then cx_1 is the integer horizontal coordinate to the right of cx, and cy_1 is the integer vertical coordinate below cy (cy_1 is the integer coordinate below cy in the image coordinate system). In this way, it can be determined that the four vertex integer pixel coordinates closest to the second pixel coordinate (cx, cy) are (cx_0, cy_0), (cx_1, cy_0), (cx_0, cy_1) and (cx_1, cy_1).

[0056] Furthermore, when the integer pixel coordinates are the integer pixel coordinates of the four vertices closest to the second pixel coordinate, calculating the distance weights between the second pixel coordinate and each integer pixel coordinate includes:

[0057] Determine the horizontal coordinate difference and the vertical coordinate difference between the second pixel coordinate and the integer pixel coordinates of each vertex, and calculate the distance weight between the second pixel coordinate and the corresponding integer pixel coordinate of the vertex based on the horizontal coordinate difference and the vertical coordinate difference.

[0058] Based on the four vertex integer pixel coordinates determined above, by determining the horizontal and vertical coordinate differences between the second pixel coordinate and the integer pixel coordinates of each vertex, the product of the horizontal and vertical coordinate differences can be determined as the distance weight. The distance weight represents the "interpolation weight" of the corresponding coordinate in image A relative to the surrounding integer coordinates of the pixel coordinate (i, j) in image B after optical flow offset. These weights reflect the relative distance or contribution between the floating-point coordinate and the four surrounding integer coordinates. The distance weight is used to mix the pixel values ​​of the surrounding integer coordinates in a certain proportion to estimate the optical flow value at the floating-point coordinate. The size of the weight depends on the horizontal and vertical distance between the floating-point coordinate and each integer coordinate.

[0059] Therefore, based on the second pixel coordinate and the integer pixel coordinates of each vertex, the distance weight is calculated:

[0060] w_tl = (cx_1 - cx) * (cy_1 - cy), which represents the distance weight of the integer pixel coordinates of the upper left corner of the second pixel coordinate;

[0061] w_tr = (cx - cx_0) * (cy_1 - cy), which represents the distance weight of the integer pixel coordinates of the upper right corner of the second pixel coordinate;

[0062] w_bl = (cx_1 - cx) * (cy - cy_0), representing the distance weight of the integer pixel coordinates of the lower left corner of the second pixel coordinate;

[0063] w_br=(cx-cx_0)*(cy-cy_0), which represents the distance weight of the integer pixel coordinates of the lower right corner vertex of the second pixel coordinate.

[0064] It should be noted that in actual applications, distance weights can also be calculated based on the traversed integer pixel coordinates by traversing the adjacent integer pixel coordinates in the four directions of the second pixel coordinate. When allocating optical flow, the optical flow of the second pixel coordinate is then allocated to the integer pixel coordinate based on the distance weights. The method for calculating distance weights based on the determined integer pixel coordinates refers to the method for calculating vertex pixel coordinates described above. This application does not impose a fixed restriction on the specific integer pixel coordinates and will not be further described here.

[0065] S130. Determine, based on the distance weight and the inverse value of the first pixel coordinate, weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image, and calculate the optical flow of a second optical flow map based on the weighted optical flow information. The second optical flow map is used to store position offset information of each pixel coordinate in the second image in the first image.

[0066] Furthermore, using vertex integer pixel coordinates as an example, the offset information of the second pixel coordinate in the first image can be determined by combining it with the inverse value of the first pixel coordinate. Since this offset information is not an integer in the optical flow map, it is necessary to combine it with the distance weight to determine the weighted optical flow information of the four vertex integer pixel coordinates by splitting this offset information.

[0067] Determining weighted optical flow information of integer pixel coordinates of each vertex corresponding to the second pixel coordinate in the first image according to the distance weight and the inverse value of the first pixel coordinate includes:

[0068] The product of the inverse value of the first pixel coordinate and the distance weight is calculated as the weighted optical flow information of the integer pixel coordinate corresponding to the distance weight in the first image.

[0069] Based on the inverse value of the first pixel coordinate -F[i,j], and the distance weight, we can get:

[0070] G[cy_0,cx_0]+=-F[i,j]*w_tl;

[0071] G[cy_0,cx_1]+=-F[i,j]*w_tr

[0072] G[cy_1,cx_0]+=-F[i,j]*w_bl

[0073] G[cy_1,cx_1]+=-F[i,j]*w_br

[0074] G[cy_0,cx_0], G[cy_0,cx_1], G[cy_1,cx_0], and G[cy_1,cx_1] respectively represent the contribution of the second pixel coordinate (floating-point coordinate) to the integer pixel coordinates of the four vertices.

[0075] At the same time, corresponding to the integer coordinate points of the above vertices, the distance weights are stored:

[0076] T[cy_0,cx_0]+=wt_tl

[0077] T[cy_0,cx_1]+=wt_tr

[0078] T[cy_1,cx_0]+=wt_bl

[0079] T[cy_1,cx_1]+=wt_br

[0080] The optical flow of the second optical flow map is calculated based on the weighted optical flow information, including:

[0081] Each integer pixel coordinate in the second optical flow map is traversed, and weight normalization processing is performed based on the integer pixel coordinates and corresponding weighted optical flow information to determine the optical flow of each integer pixel coordinate in the second optical flow map.

[0082] Based on the weighted optical flow information of each integer pixel coordinate that has been determined, by traversing each integer pixel coordinate of the first optical flow map, traversing the [0, H-1] column for the horizontal coordinate i and traversing the [0, W-1] column for the vertical coordinate j, for each integer pixel coordinate and its corresponding weighted optical flow information, weight normalization processing is performed as follows:

[0083] G[i,j]=G[i,j] / T[i,j], if T[i,j]! =0

[0084] Finally, according to the above algorithm, the forward optical flow G calculated based on the reverse optical flow F can be calculated as follows: Figure 7 As shown. Using the forward optical flow G, the image B is distorted and the image A can be restored. ′ like Figure 4 As shown. You can see A ′ A high similarity with A indicates that the forward optical flow determined in this way has achieved a high accuracy.

[0085] In this way, by traversing the optical flow at each coordinate in the reverse optical flow graph F, the corresponding coordinate in G is found, and the inverse of the optical flow (i.e., the reverse motion information) is recorded in the corresponding coordinate in G. Since optical flow information is not necessarily an integer, the algorithm needs to integerize the floating-point optical flow. To maintain accuracy, the necessary weighting is required during the processing to accurately calculate the forward optical flow G.

[0086] For example, in the actual calculation process, assume i,j = 2,3, indicating the 2nd row and 3rd column. The reverse optical flow coordinates F[2,3] = (1.5, 3.5). That is, F[2,3,0] = 1.5, F[2,3,1] = 3.5. Among them, F[i,j,0] stores the horizontal optical flow, and F[i,j,1] stores the vertical optical flow. Based on the above equation:

[0087]

[0088] We can get B[2,3]=A[2+F[2,3,1],3+F[2,3,0]]=A[2+3.5,3+1.5]=A[5.5,4.5].

[0089] This means that the pixel at B[2,3] in image A has moved to A[5.5,4.5]. The coordinate offsets of 1.5, 3.5 are stored in F[2,3]. The optical flow stored in F is based on the coordinates of image B. Accordingly, the optical flow stored in G needs to be based on the coordinates of image A.

[0090] If the coordinates of A are used as the reference, B[2,3]=A[5.5,4.5] will be transformed into A[5.5,4.5]=B[2,3]. Further deduction, A[5.5,4.5]=B[5.5+G[5.5,4.5,1],4.5+G[5.5,4.5,0]]=B[2,3];

[0091] Since 5.5+G[5.5,4.5,1]=2, G[5.5,4.5,1]=-3.5;

[0092] 4.5+G[5.5,4.5,0]=3, so G[5.5,4.5,0]=-1.5.

[0093] Therefore, F[2,3]=(1.5,3.5) optical flow corresponds to G[5.5,4.5]=(-1.5,-3.5) in G, which means that the pixel of A[5.5,4.5] is moved to the position of B[2,3] in B.

[0094] However, A[5.5,4.5] is not an integer pixel coordinate and does not exist. Therefore, the optical flow of (-1.5,-3.5) needs to be dispersed and stored in the four adjacent integer coordinates of (5.5,4.5). Based on the above weighted optical flow information calculation formula, we can get:

[0095] G[5,4]+=(-1.5,-3.5)*0.25

[0096] G[5,5]+=(-1.5,-3.5)*0.25

[0097] G[6,4]+=(-1.5,-3.5)*0.25

[0098] G[6,5]+=(-1.5,-3.5)*0.25

[0099] T[5,4]+=0.25

[0100] T[5,5]+=0.25

[0101] T[6,4]+=0.25

[0102] T[6,5]+=0.25

[0103] Finally, normalize by weight:

[0104] A possible calculation method is: G[6,5]=G[6,5] / T[6,5]=(-1.5,-3.5)*0.25 / 0.25=(-1.5,-3.5)

[0105] Normalization is done because G[6,5] may be superimposed with more optical flows during other traversals, so a weighted average is required. For example, when i,j loops to (3,4), we find that F[3,4] = (1.5,2.8)

[0106] After inverting the optical flow, we get G[5.8,5.5] = (-1.5,-2.8). At this point, G[6,5] + = (-1.5,-2.8)*(0.8*0.5). Therefore, the value of G[6,5] is (-1.5,-3.5)*0.25+(-1.5,-2.8)*0.4=(-0.375,-0.875)+(-0.6,-1.12)=(-0.975,-1.987). The value of T[6,5] is 0.25+0.4=0.65. During the second loop traversal, we get G[6,5] = G[6,5] / T[6,5] = (-0.975,-1.987) / 0.65=(-1.5,-3.0569).

[0107] For other integer pixel coordinates adjacent to the second pixel coordinate, after determining the distance weight, the weighted optical flow information of the integer pixel coordinates in the first image is calculated similarly with reference to the above calculation method, and then the second optical flow map is determined based on the weighted optical flow information.

[0108] Based on this method, the optical flow map in one direction between two images can be quickly calculated from the optical flow map in the other direction. This can be used to reduce the time overhead of the interpolation algorithm. In the beauty algorithm, after completing the deformation using the reverse optical flow, the forward optical flow can be used to quickly query the deformed position of key points.

[0109] Among them, in live broadcast apps or video apps, the video frame rate received by the audience is low. At this time, the frame rate can be increased by the interpolation algorithm to improve the user experience. During the operation of the interpolation algorithm, a new frame needs to be inserted between two adjacent video frames A and B. After calculating the forward optical flow from A to B, the image optical flow determination method based on this application can quickly solve the reverse optical flow without recalculating the reverse optical flow from B to A, thereby accelerating the interpolation algorithm.

[0110] In addition, in the beautification algorithm, the overall shape of the face and the position and size of the facial features will be adjusted. Some deformation algorithms are implemented through optical flow. Based on the given image A, the algorithm is used to directly calculate the reverse optical flow, and the image A is deformed to obtain image B. At this time, the key points of A need to be deformed as well, otherwise it will affect the processes that rely on key points after beautification, such as facial special effects. In this case, the image optical flow determination method based on the present application can calculate the forward optical flow based on the reverse optical flow, and find the position of the key point of A after deformation in the forward optical flow as the new key point position for use in subsequent processes.

[0111] In the above, a first optical flow map is obtained, and the second pixel coordinates corresponding to the first pixel coordinates of the first image in the second image are determined based on the first optical flow map. The first optical flow map is used to store the position offset information of each pixel coordinate in the first image in the second image; the integer pixel coordinates adjacent to the second pixel coordinate are determined, and the distance weights between the second pixel coordinate and each integer pixel coordinate are calculated; the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image is determined based on the distance weights and the inverse value of the first pixel coordinate, and the optical flow of the second optical flow map is calculated based on the weighted optical flow information. The second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image. By using the above technical means, the second pixel coordinates corresponding to the first pixel coordinate in the second image are determined based on the first optical flow map, the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate is determined in the first image, and the optical flow of the second optical flow map is calculated based on the weighted optical flow information. In this way, the optical flow map in one direction between the two images can be quickly calculated based on the optical flow map in the other direction, simplifying the optical flow calculation process, avoiding the situation where the calculation of two optical flow maps separately leads to high computational complexity, improving the efficiency of optical flow calculation, and thus improving the efficiency of image processing.

[0112] Based on the above embodiments, Figure 7 This is a schematic diagram of the structure of an image optical flow determination system provided by this application. Figure 7 The image optical flow determination system provided in this embodiment specifically includes: a coordinate determination module 21, a weight calculation module 22 and an optical flow calculation module 23.

[0113] The coordinate determination module 21 is configured to obtain a first optical flow map, and determine, based on the first optical flow map, a second pixel coordinate corresponding to a first pixel coordinate of the first image in the second image, wherein the first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image;

[0114] The weight calculation module 22 is configured to determine integer pixel coordinates adjacent to the second pixel coordinate and calculate distance weights between the second pixel coordinate and each integer pixel coordinate;

[0115] The optical flow calculation module 23 is configured to determine the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image based on the distance weight and the inverse value of the first pixel coordinate, and calculate the optical flow of the second optical flow map based on the weighted optical flow information. The second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image.

[0116] Specifically, determining, according to the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image includes:

[0117] Obtaining horizontal axis offset information and vertical axis offset information stored in the first pixel coordinate of the first image corresponding to the first optical flow map;

[0118] A second pixel coordinate corresponding to the first pixel coordinate in the second image is calculated based on the first pixel coordinate, the horizontal axis offset information, and the vertical axis offset information.

[0119] Specifically, determining integer pixel coordinates adjacent to the second pixel coordinate includes:

[0120] The horizontal axis coordinate value and the vertical axis coordinate value of the second pixel coordinate are rounded down, and the integer pixel coordinates of the four vertices closest to the second pixel coordinate are determined as the integer pixel coordinates according to the rounding results.

[0121] Calculating the distance weights between the second pixel coordinate and each integer pixel coordinate includes:

[0122] Determine the horizontal coordinate difference and the vertical coordinate difference between the second pixel coordinate and the integer pixel coordinates of each vertex, and calculate the distance weight between the second pixel coordinate and the corresponding integer pixel coordinate of the vertex based on the horizontal coordinate difference and the vertical coordinate difference.

[0123] Specifically, determining weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image according to the distance weight and the inverse value of the first pixel coordinate includes:

[0124] The product of the inverse value of the first pixel coordinate and the distance weight is calculated as the weighted optical flow information of the vertex integer pixel coordinate corresponding to the distance weight in the first image.

[0125] Calculating the optical flow of the second optical flow map based on the weighted optical flow information includes:

[0126] Each integer pixel coordinate in the second optical flow map is traversed, and weight normalization processing is performed based on the integer pixel coordinates and corresponding weighted optical flow information to determine the optical flow of each integer pixel coordinate in the second optical flow map.

[0127] In the above, a first optical flow map is obtained, and the second pixel coordinates corresponding to the first pixel coordinates of the first image in the second image are determined based on the first optical flow map. The first optical flow map is used to store the position offset information of each pixel coordinate in the first image in the second image; the integer pixel coordinates adjacent to the second pixel coordinate are determined, and the distance weights between the second pixel coordinate and each integer pixel coordinate are calculated; the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image is determined based on the distance weights and the inverse value of the first pixel coordinate, and the optical flow of the second optical flow map is calculated based on the weighted optical flow information. The second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image. By using the above technical means, the second pixel coordinates corresponding to the first pixel coordinate in the second image are determined based on the first optical flow map, the weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate is determined in the first image, and the optical flow of the second optical flow map is calculated based on the weighted optical flow information. In this way, the optical flow map in one direction between the two images can be quickly calculated based on the optical flow map in the other direction, simplifying the optical flow calculation process, avoiding the situation where the calculation of two optical flow maps separately leads to high computational complexity, improving the efficiency of optical flow calculation, and thus improving the efficiency of image processing.

[0128] The image optical flow determination system provided in the embodiment of the present application can be configured to execute the image optical flow determination method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0129] Based on the above practical example, the present application embodiment also provides an image optical flow determination device, referring to Figure 8 , the image optical flow determination device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the image optical flow determination method described in any embodiment of the present application (for example, the coordinate determination module, weight calculation module and optical flow calculation module in the image optical flow determination system). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizes the above-mentioned image optical flow determination method. The input device can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device may include a display device such as a display screen. The image optical flow determination device provided above can be configured to execute the image optical flow determination method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0130] Based on the above embodiments, embodiments of the present application further provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When executed by a computer processor, the computer-executable instructions are configured to execute a method for determining image optical flow. The storage medium can be any of various types of memory devices or storage devices. Of course, the computer-executable instructions of the non-volatile computer-readable storage medium provided in embodiments of the present application are not limited to the method for determining image optical flow described above, but can also execute related operations in the method for determining image optical flow provided in any embodiment of the present application.

[0131] Based on the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the image optical flow determination method described in each embodiment of the present application.

Claims

1. A method for determining image optical flow, characterized in that: include: Acquire a first optical flow map, and determine, according to the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image, wherein the first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image; Determine integer pixel coordinates adjacent to the second pixel coordinate, and calculate distance weights between the second pixel coordinate and each of the integer pixel coordinates; According to the distance weight and the inverse value of the first pixel coordinate, weighted optical flow information of each of the integer pixel coordinates corresponding to the second pixel coordinate in the first image is determined, and the optical flow of a second optical flow map is calculated based on the weighted optical flow information, where the second optical flow map is used to store position offset information of each pixel coordinate in the second image in the first image.

2. The method for determining image optical flow according to claim 1, characterized in that: The determining, according to the first optical flow map, second pixel coordinates corresponding to the first pixel coordinates of the first image in the second image includes: Obtaining horizontal axis offset information and vertical axis offset information stored in the first pixel coordinates of the first image corresponding to the first optical flow map; A second pixel coordinate corresponding to the first pixel coordinate in the second image is calculated based on the first pixel coordinate, the horizontal axis offset information, and the vertical axis offset information.

3. The method for determining image optical flow according to claim 1, characterized in that: The determining of integer pixel coordinates adjacent to the second pixel coordinate comprises: The horizontal axis coordinate value and the vertical axis coordinate value of the second pixel coordinate are rounded down, and the integer pixel coordinates of four vertices closest to the second pixel coordinate are determined according to the rounding result as the integer pixel coordinate.

4. The method for determining image optical flow according to claim 3, characterized in that: The calculating the distance weight between the second pixel coordinate and each of the integer pixel coordinates includes: Determine a horizontal axis coordinate difference and a vertical axis coordinate difference between the second pixel coordinate and each of the vertex integer pixel coordinates, and calculate a distance weight between the second pixel coordinate and the corresponding vertex integer pixel coordinate based on the horizontal axis coordinate difference and the vertical axis coordinate difference.

5. The method for determining image optical flow according to claim 1, characterized in that: The determining, according to the distance weight and the inverse value of the first pixel coordinate, weighted optical flow information of each integer pixel coordinate corresponding to the second pixel coordinate in the first image includes: The product of the inverse value of the first pixel coordinate and the distance weight is calculated as the weighted optical flow information of the integer pixel coordinate corresponding to the distance weight in the first image.

6. The method for determining image optical flow according to claim 1, characterized in that: The calculating the optical flow of the second optical flow map based on the weighted optical flow information includes: Each integer pixel coordinate in the second optical flow map is traversed, and weight normalization processing is performed based on the integer pixel coordinates and the corresponding weighted optical flow information to determine the optical flow of each integer pixel coordinate in the second optical flow map.

7. An image optical flow determination system, characterized in that: include: a coordinate determination module configured to obtain a first optical flow map, and determine, according to the first optical flow map, second pixel coordinates corresponding to first pixel coordinates of the first image in the second image, wherein the first optical flow map is used to store position offset information of each pixel coordinate in the first image in the second image; a weight calculation module, configured to determine integer pixel coordinates adjacent to the second pixel coordinate, and calculate distance weights between the second pixel coordinate and each of the integer pixel coordinates; The optical flow calculation module is configured to determine the weighted optical flow information of each of the vertex integer pixel coordinates corresponding to the second pixel coordinate in the first image according to the distance weight and the inverse value of the first pixel coordinate, and calculate the optical flow of the second optical flow map based on the weighted optical flow information, wherein the second optical flow map is used to store the position offset information of each pixel coordinate in the second image in the first image.

8. An image optical flow determination device, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image optical flow determination method as described in any one of claims 1-6.

9. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer processor, the computer-executable instructions are configured to execute the image optical flow determination method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the image optical flow determination method according to any one of claims 1 to 6.

Images