Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium, and the method comprises the steps: carrying out the optical flow calculation of a first image and a second image collected by a same image collection module, determining the first optical flow data corresponding to a first pixel point in the second image, and enabling the first image to be the previous frame image of the second image; according to the coordinates of the first optical flow data, multiple pieces of second optical flow data are determined, and the coordinates of at least one piece of optical flow data in the multiple pieces of second optical flow data and the coordinates of the first optical flow data have offset which is not equal to zero in the first direction and the second direction; determining a correction value of the first optical flow data according to the first optical flow data and the plurality of second optical flow data; and correcting the first optical flow data by using the correction value to obtain target optical flow data corresponding to the first pixel point. Sub-pixel-level optical flow calculation can be realized, and the precision of optical flow data can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, electronic device, chip, and storage medium. Background Art

[0002] In the field of image processing, image optical flow is used to indicate the motion pattern of pixels in an image. Traditional methods can obtain dense optical flow data at the pixel level to estimate the motion of each pixel in the image. Motion analysis of each pixel can be performed in various complex scenes, which can provide a good foundation for subsequent image processing tasks such as video stitching and fusion. However, the current dense optical flow data acquisition method cannot obtain dense optical flow data at the sub-pixel level, resulting in poor accuracy of dense optical flow. Summary of the Invention

[0003] The present disclosure provides an image processing method, device, electronic device, chip and storage medium to solve the problems in the related art.

[0004] A first aspect embodiment of the present disclosure proposes an image processing method, the method comprising: performing optical flow calculation on a first image and a second image captured by the same image acquisition module, determining first optical flow data corresponding to a first pixel point in the second image, where the first image is a previous frame image of the second image; determining a plurality of second optical flow data based on the coordinates of the first optical flow data, wherein the coordinates of at least one of the plurality of second optical flow data are offset from the coordinates of the first optical flow data in both a first direction and a second direction and are not equal to zero; determining a correction value of the first optical flow data based on the first optical flow data and the plurality of second optical flow data; and correcting the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel point.

[0005] In some embodiments of the present disclosure, determining multiple second optical flow data based on first optical flow data includes: determining the maximum optical flow number in the first direction and the maximum optical flow number in the second direction; determining at least one offset of the first optical flow data in the first direction and / or the second direction based on the maximum optical flow number in the first direction and the maximum optical flow number in the second direction; determining multiple second optical flow data based on the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

[0006] In some embodiments of the present disclosure, determining multiple second optical flow data based on the coordinates of first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction includes: determining the maximum number of optical flows in the first direction to be N, and the maximum number of optical flows in the second direction to be M; offsetting the coordinates of the first optical flow data along the first direction based on the N-1 offsets of the first optical flow data in the first direction to obtain N-1 second optical flow data; offsetting the coordinates of the first optical flow data along the second direction based on the M-1 offsets of the first optical flow data in the second direction to obtain M-1 second optical flow data; offsetting the coordinates of the first optical flow data along the first direction and the second direction based on the N-1 offsets of the first optical flow data in the first direction and the M-1 offsets in the second direction to obtain N×MN-M+1 second optical flow data.

[0007] In some embodiments of the present disclosure, determining a correction value of the first optical flow data based on the first optical flow data and the multiple second optical flow data includes: determining a first generation value corresponding to the first optical flow data, and multiple second generation values ​​corresponding to the multiple second optical flow data; determining a cost fitting equation based on the first generation value, the multiple second generation values, the first optical flow data, and the multiple second optical flow data, the cost fitting equation being used to indicate the correspondence between the multiple optical flow data and the cost value of the first pixel point; and determining the correction value of the first optical flow data based on the cost fitting equation.

[0008] In some embodiments of the present disclosure, using a correction value to correct the first optical flow data to obtain target optical flow data corresponding to the first pixel point includes: determining a coefficient matrix based on a cost fitting equation; determining a first correction value corresponding to the first optical flow data in the first direction and a second correction value corresponding to the second direction based on the coefficient matrix; using the first correction value to correct the first optical flow value of the first optical flow data in the first direction, and using the second correction value to correct the second optical flow value of the first optical flow data in the second direction; and determining the target optical flow data based on the corrected first optical flow value and the corrected second optical flow value.

[0009] In some embodiments of the present disclosure, the method also includes: when the first correction value or the second correction value cannot be determined according to the coefficient matrix, determining the first optical flow data as the target optical flow data; when the absolute value of the first correction value is greater than or equal to the first value, or the absolute value of the second correction value is greater than or equal to the first value, determining the first optical flow data as the target optical flow data.

[0010] A second aspect embodiment of the present disclosure proposes an image processing device, which includes: a first processing unit, used to perform optical flow calculation on a first image and a second image captured by the same image acquisition module, and determine first optical flow data corresponding to a first pixel point in the second image, where the first image is a previous frame image of the second image; a second processing unit, used to determine a plurality of second optical flow data based on the coordinates of the first optical flow data, where the coordinates of at least one of the plurality of second optical flow data are offset from the coordinates of the first optical flow data in both the first and second directions; a third processing unit, used to determine a correction value of the first optical flow data based on the first optical flow data and the plurality of second optical flow data; and a fourth processing unit, used to correct the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel point.

[0011] In some embodiments of the present disclosure, the second processing unit is further used to: determine the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction; determine at least one offset of the first optical flow data in the first direction and / or the second direction based on the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction; determine multiple second optical flow data based on the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

[0012] In some embodiments of the present disclosure, the third processing unit is also used to: determine a first generation value corresponding to the first optical flow data, and multiple second generation values ​​corresponding to the multiple second optical flow data; determine a cost fitting equation based on the first generation value, the multiple second generation values, the first optical flow data, and the multiple second optical flow data, the cost fitting equation being used to indicate the correspondence between the multiple optical flow data and the cost value of the first pixel point; determine a correction value of the first optical flow data based on the cost fitting equation.

[0013] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect embodiment of the present disclosure.

[0014] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0015] The fifth aspect embodiment of the present disclosure proposes a chip, characterized in that it includes at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, and the processor communicates with the communication interface and implements the method described in the first aspect embodiment of the present disclosure through logic circuits or executing code instructions.

[0016] In summary, the image processing method proposed in the present disclosure can determine, based on the first optical flow data of the first pixel point, multiple second optical flow data that have an offset not equal to zero with the first optical flow data in the first direction and / or the second direction, and determine a correction value based on the first optical flow data and the multiple second optical flow data. Thereafter, the correction value is used to correct the first optical flow data, thereby optimizing the first optical flow data and calculating sub-pixel optical flow data, thereby improving the accuracy of the optical flow data.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0019] Figure 1 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 1 ;

[0020] Figure 2 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 2 ;

[0021] Figure 3 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 3 ;

[0022] Figure 4 A schematic flow chart of a polynomial fitting sub-pixel calculation method for dense motion estimation provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic structural diagram of an image processing device provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of the chip structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0027] In image processing, image optical flow is used to indicate the motion patterns of pixels within an image. Traditionally, semi-global matching (SGM) has been used to calculate dense optical flow between two images. This method propagates cost values ​​in multiple directions (4 / 8 / 16) and uses a Winner Takes All (WTA) strategy to obtain optical flow values, thereby approximating the globally optimal dense optical flow information. However, this method is only accurate to integers and cannot obtain sub-pixel dense optical flow. When applied to scenarios such as local image alignment, pixel-level accuracy is insufficient because dense optical flow is calculated on lower-resolution images, while warping is performed at higher resolutions. Higher-precision sub-pixel optical flow results are required.

[0028] Alternatively, in one implementation, sub-pixel calculations can be performed separately in the x- and y-directions. This solution calculates the optical flow in the x- and y-directions separately. For the x-direction optical flow, only the cost values ​​at the three left and right positions are fitted using a quadratic function. The lowest cost value in the x-direction is obtained as the final output sub-pixel optical flow value in the x-direction. The same applies to the y-direction. When calculating the sub-pixel optical flow, this solution only considers the distribution of cost values ​​in the specific left-right and up-down directions, and fails to simultaneously consider the distribution of cost values ​​in other directions, resulting in poor performance.

[0029] Therefore, to address the above issues, this disclosure proposes an image processing method. The method described in this disclosure can be applied to image signal processor (ISP) chips and combined with a dense optical flow calculation solution. It can also be implemented using a software solution. This method can obtain more accurate sub-pixel dense optical flow.

[0030] The specific contents of this method are as follows.

[0031] Figure 1 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 1 .like Figure 1 As shown, the method may include the following steps.

[0032] Step 101 : performing optical flow calculation on a first image and a second image acquired by the same image acquisition module to determine first optical flow data corresponding to a first pixel in the second image.

[0033] In some embodiments, the first image is the previous frame image of the second image. Optionally, pixel-level optical flow data of each pixel point on the second image can be calculated based on the first image and the second image. Optionally, for a pixel point on the second image, the position of the pixel point on the first image can be determined, and then the optical flow data of the pixel point can be determined based on the position of the pixel point on the first image and the position of the pixel point on the second image, wherein the optical flow data is used to indicate the motion pattern of the pixel point from the second image to the first image. Optionally, the optical flow data can be a vector, which can include the movement direction and speed of the pixel point between the second image and the first image.

[0034] Optionally, the first optical flow data is dense optical flow data, which is obtained by calculating the motion vector of each pixel in the image. Optionally, the first optical flow data includes a first optical flow value in a first direction and a second optical flow value in a second direction, where the first direction can be the horizontal x direction and the second direction can be the vertical y direction. That is, the first optical flow value is used to indicate the distance the first pixel moves in the horizontal direction when it moves from a position on the second image to a position on the first image, and the second optical flow value is used to indicate the distance the first pixel moves in the vertical direction when it moves from a position on the second image to a position on the first image.

[0035] Optionally, the first optical flow data can be expressed as (u, v), then the first optical flow value is u, and the second optical flow value is v. For example, the coordinates of the first pixel point on the first image are (3, 5), and the coordinates of the first pixel point on the second image are (4, 4), then the first optical flow data is (-1, 1), then the first optical flow value is -1, and the second optical flow value is 1.

[0036] In some embodiments, optionally, a semi-global matching (SGM) method can be used to calculate the dense optical flow between the two images to determine the first optical flow data corresponding to each pixel point on the second image, where the first pixel point is any pixel point among multiple pixel points on the second image. Optionally, each pixel point can correspond to multiple optical flow data, and the first optical flow data is the most accurate optical flow data among the multiple optical flow data corresponding to the first pixel point. The most accurate accuracy can be the highest similarity or the smallest cost.

[0037] For example, after determining multiple optical flow data of the first pixel point, the first pixel point in the second image can be mapped to the first position in the first image according to the multiple optical flow data. Then, the similarity or cost value corresponding to the first pixel point can be determined based on the difference between the first position and the first pixel point. Then, the first optical flow data can be determined based on the similarity or cost value. The higher the similarity, the lower the cost value. Optionally, the similarity or cost value corresponding to the first pixel point can be determined based on the difference in brightness. Optionally, the cost value can be an aggregated cost value, a superimposed cost value, etc. For example, the SGM method can be used to transfer the cost value in multiple directions to obtain an aggregated cost value, which is not limited by the present disclosure.

[0038] Step 102: Determine a plurality of second optical flow data according to the coordinates of the first optical flow data.

[0039] In some embodiments, the multiple second optical flow data are optical flow data of the first pixel point. Optionally, the multiple optical flow data of the first pixel point can be used to indicate the motion vector of different positions of the first pixel point. The coordinates of at least one of the multiple second optical flow data have an offset that is not equal to zero with the coordinates of the first optical flow data in both the first direction and the second direction.

[0040] In some embodiments, the coordinates of multiple second optical flow data are different, and the coordinates of the second optical flow data are different from the coordinates of the first optical flow data. Optionally, the first optical flow data can be used as the center point, that is, the coordinates of the first optical flow data can be expressed as (0, 0), and the coordinates of the second optical flow data can be determined based on the positional relationship with the first optical flow data. For example, the offset of the second optical flow data from the first optical flow data in the x direction is 1, and there is no offset in the y direction with the first optical flow data, then the coordinates of the second optical flow data can be expressed as (1, 0), that is, the coordinates of the second optical flow data can be relative coordinates relative to the first optical flow data.

[0041] In some embodiments, the coordinates of at least one optical flow data among the multiple second optical flow data are offset from the coordinates of the first optical flow data in both the first direction and the second direction, that is, the position of the second optical flow data is offset from the first optical flow data in both the x direction and the y direction, and the offset is not equal to 0.

[0042] Step 103: Determine a correction value of the first optical flow data according to the first optical flow data and the plurality of second optical flow data.

[0043] In some embodiments, a correction value of the first optical flow data can be determined based on the first optical flow data and multiple second optical flow data. Among the multiple second optical flow data, there is at least one optical flow data whose coordinates are offset from the coordinates of the first optical flow data in both the first and second directions and are not equal to zero. That is, not only optical flow data that is offset from the first optical flow data in the x direction or the y direction can be used, but also optical flow data that is offset from the first optical flow data in both the x direction and the y direction can be used to determine the correction value. This can achieve the purpose of determining the correction value using optical flow data in multiple directions around the first optical flow data, and ensure the accuracy of the correction value.

[0044] For example, when the number of second optical flow data is 8, that is, with the first optical flow data as the center, the other optical flow data in the surrounding 3×3 squares are the second optical flow data. At this time, the second optical flow data in the corners of the 3×3 squares and the first optical flow data at the center are offset in both the x and y directions. At this time, the multiple second optical flow data include not only the optical flow data in the x and y directions of the first optical flow data, but also the optical flow data at a 45-degree angle. It is possible to determine the correction value using the optical flow data in multiple directions around the first optical flow data, thereby ensuring the accuracy of the correction value.

[0045] Step 104 : Correct the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel.

[0046] In some embodiments, the first optical flow data can be corrected using a correction value. Optionally, the correction value can include a first correction value Δu corresponding to the first optical flow data in the first direction, and a second correction value Δv corresponding to the second direction. The first correction value Δu can then be used to correct the first optical flow value x of the first optical flow data in the first direction. Optionally, the first optical flow value and the first correction value can be added to obtain a corrected first optical flow value. Similarly, the second correction value Δv can be used to correct the second optical flow value y of the first optical flow data in the second direction. Optionally, the second optical flow value and the second correction value can be added to obtain a corrected second optical flow value. The target optical flow data can then be determined based on the corrected first optical flow value and the second optical flow value. The target optical flow data includes the corrected first optical flow value and the second optical flow value.

[0047] In summary, the above-mentioned embodiments of the present disclosure can determine, based on the first optical flow data of the first pixel point, multiple second optical flow data that have an offset not equal to zero with the first optical flow data in the first direction and / or the second direction, and determine a correction value based on the first optical flow data and the multiple second optical flow data. Thereafter, the correction value is used to correct the first optical flow data, thereby optimizing the first optical flow data and calculating sub-pixel optical flow data, thereby improving the accuracy of the optical flow data.

[0048] Figure 2 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 2 .like Figure 2 As shown, based on Figure 1 In the illustrated embodiment, the method includes the following steps.

[0049] Step 201: Determine the maximum optical flow quantity in the first direction and the maximum optical flow quantity in the second direction.

[0050] In some embodiments, the maximum number of optical flows N in the first direction and the maximum number of optical flows M in the second direction can be determined. For example, the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction can be preset, or the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction can be determined based on the accuracy of the indicated optical flow data, etc., and the present disclosure is not limited to this. Optionally, the values ​​of N and M can be the same or different, and N and M are positive integers greater than 0.

[0051] Step 202 : Determine at least one offset of the first optical flow data in the first direction and / or the second direction according to the maximum optical flow quantity in the first direction and the maximum optical flow quantity in the second direction.

[0052] In some embodiments, at least one offset of the first optical flow data in the first direction and / or the second direction can be determined based on the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction, and the at least one offset is used to determine a plurality of second optical flow data. Optionally, the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction may have a corresponding relationship with the at least one offset, that is, when the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction are determined, the value of the at least one offset is fixed. Optionally, the corresponding relationship may be a preset corresponding relationship.

[0053] In some embodiments, at least one offset may be an offset relative to the coordinates of the first optical flow data, wherein the at least one offset may include an offset in a first direction, i.e., an offset in the x-direction, and may also include an offset in a second direction, i.e., an offset in the y-direction. Optionally, the offset in the first direction and the offset in the second direction may be determined based on the maximum optical flow quantity in the first direction and the maximum optical flow quantity in the second direction. For example, when N and M are both 3, the offset in the x-direction may include -1, 0, and 1, and the offset in the y-direction may include -1, 0, and 1. For another example, when N is 3 and y is 4, the offset in the x-direction may include -1, 0, and 1, and the offset in the y-direction may include -1, 0, 1, and 2, and so on. The specific correspondence is not limited in this disclosure.

[0054] Step 203 : Determine a plurality of second optical flow data according to the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

[0055] In some embodiments, determining multiple second optical flow data based on the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction includes: determining the maximum number of optical flows in the first direction to be N, and the maximum number of optical flows in the second direction to be M; offsetting the coordinates of the first optical flow data along the first direction according to the N-1 offsets of the first optical flow data in the first direction to obtain N-1 second optical flow data; offsetting the coordinates of the first optical flow data along the second direction according to the M-1 offsets of the first optical flow data in the second direction to obtain M-1 second optical flow data; offsetting the coordinates of the first optical flow data along the first direction and the second direction according to the N-1 offsets of the first optical flow data in the first direction and the M-1 offsets in the second direction to obtain N×MN-M+1 second optical flow data.

[0056] Optionally, the multiple second optical flow data may include second optical flow data that is offset only from the first optical flow data in the first direction. For example, N and M are both 3, the coordinate of the first optical flow data is expressed as (0, 0), the offset in the x-direction may include -1, 0 and 1, and the offset in the y-direction may include -1, 0 and 1. Then the coordinates of the second optical flow data that is offset only from the first optical flow data in the first direction are (-1, 0) and (1, 0) respectively. That is, after determining the offset, the coordinates of the first optical flow data can be offset according to the offset to obtain the coordinates of at least one second optical flow data. Optionally, the number of second optical flow data that is offset only from the first optical flow data in the first direction is N-1.

[0057] In some embodiments, there is second optical flow data among the multiple second optical flow data that is offset from the first optical flow data only in the second direction. For example, N and M are both 3, the coordinate of the first optical flow data is expressed as (0, 0), the offset in the x-direction may include -1, 0 and 1, and the offset in the y-direction may include -1, 0 and 1. Then the coordinates of the second optical flow data that is offset from the first optical flow data only in the second direction are (0, -1) and (0, 1) respectively, and the number of second optical flow data that is offset from the first optical flow data only in the second direction is M-1.

[0058] In some embodiments, there is second optical flow data in the plurality of second optical flow data that is offset from the first optical flow data in both the first direction and the second direction. For example, N and M are both 3, the coordinate of the first optical flow data is represented as (0, 0), the offset in the x-direction may include -1, 0 and 1, and the offset in the y-direction may include -1, 0 and 1. Then, the coordinates of the second optical flow data that are offset from the first optical flow data in both the first direction and the second direction are (-1, 1), (1, 1), (-1, -1), (1, 1), and the number of second optical flow data that are offset from the first optical flow data in both the first direction and the second direction is M-1.

[0059] In other words, after N and M are determined, the rest of the optical flow data within the N×M square can be determined as the second optical flow data with the first optical flow data as the center.

[0060] In summary, the above-mentioned embodiments of the present application can determine multiple second optical flow data that have an offset that is not equal to zero with the first optical flow data in the first direction and / or the second direction, and determine a correction value based on the first optical flow data and the multiple second optical flow data. Thereafter, the correction value is used to correct the first optical flow data, thereby optimizing the first optical flow data and calculating sub-pixel optical flow data, thereby improving the accuracy of the optical flow data.

[0061] Figure 3 A schematic diagram of an image processing method provided in an embodiment of the present disclosure Figure 3 .like Figure 3 As shown, based on Figure 1 In the illustrated embodiment, the method includes the following steps.

[0062] Step 301: Determine a first generation value corresponding to first optical flow data and a plurality of second generation values ​​corresponding to a plurality of second optical flow data.

[0063] In some embodiments, the cost value corresponding to each optical flow data can be determined separately, including a first cost value corresponding to the first optical flow data, and a plurality of second cost values ​​corresponding to the plurality of second optical flow data. After determining the plurality of optical flow data of the first pixel point, the first pixel point in the second image can be mapped to the first position in the first image according to the plurality of optical flow data. Then, the similarity or cost value corresponding to the first pixel point can be determined based on the difference between the first position and the first pixel point. Then, the first optical flow data can be determined based on the similarity or cost value. The higher the similarity, the lower the cost value. Optionally, the similarity or cost value corresponding to the first pixel point can be determined based on the difference in brightness. Optionally, the cost value can be an aggregated cost value, a superimposed cost value, etc. For example, the SGM method can be used to transfer the cost value in multiple directions to obtain an aggregated cost value, which is not limited by the present disclosure.

[0064] In some embodiments, optionally, the cost value can be determined by using the sum of absolute differences (SAD) of pixel differences. For example, a first window can be determined with the first optical flow data as the center, and the position corresponding to the first optical flow data in the first image can be determined based on the first optical flow data. Then, a second window can be determined with the position corresponding to the first optical flow data in the first image as the center, and then the cost of the pixels covered in the first window and the pixels covered in the second window can be calculated by SAD to obtain a cost value. Alternatively, the cost of the pixels covered in the first window and the pixels covered in the second window can be calculated by Census transformation and Hamming distance to obtain a cost value. This disclosure is not limited to this.

[0065] Step 302: Determine a cost fitting equation based on the first generation value, the plurality of second generation values, the first optical flow data, and the plurality of second optical flow data.

[0066] In some embodiments, multiple cost values ​​and multiple optical flow data can be fitted, that is, the first cost value, multiple second cost values, first optical flow data, and multiple second optical flow data are used for fitting to obtain a cost fitting equation. The cost fitting equation is used to indicate the correspondence between multiple optical flow data and cost values ​​of the first pixel point. Optionally, the least squares method can be used to fit multiple cost values ​​and multiple optical flow data to obtain a fitting relationship between multiple cost values ​​and multiple optical flow data.

[0067] For example, assuming that the cost value of the optical flow data satisfies the quadratic surface, let Z = a0+a1x+a2y+a3xy+a4x 2 +a5y 2 , where Z represents the cost value.

[0068] Using the least squares method for fitting, the cost fitting equation is expressed as:

[0069]

[0070] Among them, m is the number of points involved in the operation. When N is 3, 9 optical flows of 3×3 around the integer optical flow are used for fitting, so m=9.

[0071] Step 303: Determine a correction value of the first optical flow data according to the cost fitting equation.

[0072] In some embodiments, using a correction value to correct the first optical flow data to obtain target optical flow data corresponding to the first pixel point includes: determining a coefficient matrix based on a cost fitting equation; determining a first correction value corresponding to the first optical flow data in the first direction and a second correction value corresponding to the second direction based on the coefficient matrix; using the first correction value to correct the first optical flow value of the first optical flow data in the first direction, and using the second correction value to correct the second optical flow value of the first optical flow data in the second direction; and determining the target optical flow data based on the corrected first optical flow value and the corrected second optical flow value.

[0073] In some embodiments, the coefficient matrix may be determined according to the above cost fitting equation. In conjunction with the example of step 302 , the specific operations are as follows.

[0074] Derivatives a0 to a5 are taken separately and their derivatives are set to 0: We can get:

[0075]

[0076] Substituting it into matrix form, we can get:

[0077] U·A=Z

[0078] Will Omitted as ∑, then:

[0079]

[0080]

[0081] Transform the above formula to obtain the coefficient matrix A:

[0082] A=Z·U -1

[0083] In some embodiments, after the coefficient matrix is ​​determined, the first correction value Δu and the second correction value Δv can be determined according to the coefficient matrix, where (x i ,y i ) is the coordinate of 3×3 points centered on the integer optical flow, that is, (x i ,y i ) is known, z i is a known cost value, so the matrix Z and the matrix U -1 It can be obtained by calculation, thereby calculating the coefficient matrix A. After obtaining the coefficient matrix, the position of the lowest point of the surface can be calculated. The coordinates of the lowest point position are the correction values. Among them, the position of the lowest point can be expressed as (Δu, Δv). The position of the lowest point can be determined according to the following formula:

[0084]

[0085] In some embodiments, the method further includes: when the first correction value or the second correction value cannot be determined according to the coefficient matrix, determining the first optical flow data as the target optical flow data; when the absolute value of the first correction value is greater than or equal to the first value, or the absolute value of the second correction value is greater than or equal to the first value, determining the first optical flow data as the target optical flow data.

[0086] Among them, when the first correction value or the second correction value cannot be determined according to the coefficient matrix, it may be that the cost fitting equation obtained by fitting does not have a lowest position, for example, a4≤0; a5≤0, in this case the surface is an upper convex surface or a saddle surface, and the minimum value cannot be calculated; or when the first correction value or the second correction value cannot be determined according to the coefficient matrix, it may be that Δu or Δv cannot be calculated according to the above formula, for example 4a4a5-a3 2 =0, the denominator is 0 and the correction value cannot be calculated.

[0087] In some embodiments, the first value may be 1. Typically, the sub-pixel correction value should be less than 1. Therefore, when |Δu| ≥ 1 or |Δv| ≥ 1, the calculated correction value is inaccurate and cannot be used. Optionally, when the correction value cannot be determined or cannot be used during correction, the first optical flow data may be determined as the target optical flow data.

[0088] In summary, the above-mentioned embodiments of the present disclosure can determine, based on the first optical flow data of the first pixel point, multiple second optical flow data that have an offset not equal to zero with the first optical flow data in the first direction and / or the second direction, and determine a correction value based on the first optical flow data and the multiple second optical flow data. Thereafter, the correction value is used to correct the first optical flow data, thereby optimizing the first optical flow data and calculating sub-pixel optical flow data, thereby improving the accuracy of the optical flow data.

[0089] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.

[0090] The following is a polynomial fitting sub-pixel calculation method for dense motion estimation provided by an embodiment of the present disclosure. This method can solve the problem that the dense optical flow calculation solution based on SGM can only obtain pixel-level dense optical flow data. In application scenarios such as local image alignment, the pixel-level optical flow accuracy cannot meet the alignment requirements. In this method, more accurate sub-pixel dense optical flow data can be obtained. The details of this method are as follows.

[0091] like Figure 4As shown in the figure, this method uses two images and the calculated pixel-level dense optical flow data to output sub-pixel dense optical flow data. When calculating sub-pixel optical flow, the cost value of the surrounding optical flow is required. The specific number of surrounding pixels N×N can be 3×3, 4×4, 5×5, etc. In this case, 3×3 is used as an example.

[0092] Since each integer-level optical flow has a specific cost value, the goal is to obtain the sub-pixel optical flow represented by the minimum cost value in the neighborhood of the pixel-level optical flow. Therefore, the least squares method can be used to fit the polynomial. The cost value of the 3×3 area around the integer-level optical flow is fitted to obtain the coefficients of the polynomial, thereby obtaining the sub-pixel optical flow corresponding to the minimum cost value.

[0093] In dense optical flow calculations based on the SGM algorithm, each optical flow is associated with a cost value. A smaller cost value indicates greater similarity between the corresponding locations in the optical flows of the two images. The surrounding cost values ​​for the same location are generally similar. Within the neighborhood of a given optical flow, the closer it is to the correct optical flow, the greater the corresponding similarity and the smaller the cost value. Based on this assumption, we can use the cost values ​​of integer-level optical flows, combined with the cost values ​​of the neighborhood, and use quadratic surface fitting to determine the sub-pixel optical flow corresponding to the minimum cost value.

[0094] However, in areas where optical flow varies dramatically, typically where the foreground and background overlap, it's more difficult to accurately calculate the result. This is because the cost calculated for overlapping areas is based on a single block, which can belong to either the foreground or the background. This results in local discontinuities in the cost at that location, and violates the assumption that surrounding costs at the same location are similar, making it difficult to calculate accurate sub-pixel optical flow. Therefore, this solution addresses this issue by calculating sub-pixel optical flow using the cost of sub-pixel optical flow data. The specific method is as follows.

[0095] Step 1: Calculate the matrix U based on the current value of N -1 The following operations are performed on both the optical flow (u, v) and the image position (x, y):

[0096] Step 2: Calculate the cost values ​​of the surrounding 3×3 optical flows respectively. The cost value method can use SAD, census+hamming, or data cache similar to the cost value concept when using integer-level optical flow calculation. This method does not impose any restrictions.

[0097] Step 3: According to the construction of matrix Z (cost matrix), the cost value of 3×3 optical flow is used to calculate matrix Z.

[0098] Step 4: Combine matrix Z and matrix U -1Multiply them together to get matrix A (coefficient matrix).

[0099] Step 5: Get the correction values ​​Δu and Δv.

[0100] Step 6: Add the integer optical flow value to the correction value in step 5 to get the sub-pixel optical flow.

[0101] The principle derivation process of the above method is as follows.

[0102] Assuming that the cost value of the optical flow data satisfies the quadratic surface, let Z = a0+a1x+a2y+a3xy+a4x 2 +a5y 2 , where Z represents the cost value.

[0103] The least squares method is used for fitting, which is expressed as:

[0104]

[0105] Among them, m is the number of points involved in the operation. When N is 3, 9 optical flows of 3×3 around the integer optical flow are used for fitting, so m=9.

[0106] Next, we take the derivatives of a0 to a5 and set them to 0: We can get:

[0107]

[0108] Substituting it into matrix form, we can get:

[0109] U·A=Z

[0110] Will Omitted as ∑, then:

[0111]

[0112]

[0113] Transform the above formula to get

[0114] A=Z·U -1

[0115] Since (x i ,y i ) is a fixed 3×3 point centered on the integer optical flow, which can be expressed by the relative coordinates relative to the optical flow, z i is the corresponding cost value, so the matrix Z and the matrix U -1It can be obtained through calculation, thereby calculating the coefficient matrix A. After obtaining the coefficient matrix, the lowest point position of the surface can be calculated. The coordinates of the lowest point position are the offset relative to the integer-level optical flow (i.e., the correction value). The integer-level optical flow value can then be superimposed with the correction value of the fifth step to obtain the sub-pixel optical flow.

[0116]

[0117] Among them, when the following situations occur during the calculation process, it means that sub-pixel calculation cannot be performed. At this time, the correction value can be determined to be 0, and the integer-level optical flow is the sub-pixel-level optical flow.

[0118] a4≤0; a5≤0, the surface is an upper convex surface or a saddle surface, and the minimum value cannot be calculated;

[0119] 4a4a5-a3 2 =0, the denominator is 0 and the correction value cannot be calculated;

[0120] |Δu|≥1; |Δv|≥1. Since the correction value of the sub-pixel should be less than 1, the correction value calculated here is inaccurate and cannot be used.

[0121] In summary, the above examples disclosed herein require less computational effort than the original pixel-level dense optical flow calculation scheme and are easier to implement in hardware. They also improve the quality of dense optical flow and the effect of local image alignment. Compared with the previous optical flow sub-pixel scheme, the mean squared error (MSE) is smaller, and the greater the structural similarity (SSIM), the better the effect and the higher the quality.

[0122] Figure 5 FIG. 5 is a structural diagram of an image processing device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the device includes: a first processing unit 510, used to perform optical flow calculation on a first image and a second image captured by the same image acquisition module, and determine first optical flow data corresponding to a first pixel point in the second image, where the first image is a previous frame image of the second image; a second processing unit 520, used to determine a plurality of second optical flow data based on the coordinates of the first optical flow data, the plurality of second optical flow data including at least one optical flow data that meets preset conditions, and the coordinates of at least one optical flow data are offset from the coordinates of the first optical flow data in both the first direction and the second direction and are not equal to zero; a third processing unit 530, used to determine a correction value of the first optical flow data based on the first optical flow data and the plurality of second optical flow data; a fourth processing unit 540, used to correct the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel point.

[0123] In some embodiments, the second processing unit is also used to determine the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction; determine at least one offset of the first optical flow data in the first direction and / or the second direction based on the maximum number of optical flows in the first direction and the maximum number of optical flows in the second direction; determine multiple second optical flow data based on the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

[0124] In some embodiments, the second processing unit is further used to determine that the maximum number of optical flows in the first direction is N, and the maximum number of optical flows in the second direction is M; according to the N-1 offsets of the first optical flow data in the first direction, the coordinates of the first optical flow data are offset along the first direction to obtain N-1 second optical flow data; according to the M-1 offsets of the first optical flow data in the second direction, the coordinates of the first optical flow data are offset along the second direction to obtain M-1 second optical flow data; according to the N-1 offsets of the first optical flow data in the first direction and the M-1 offsets in the second direction, the coordinates of the first optical flow data are offset along the first direction and the second direction to obtain N×MN-M+1 second optical flow data.

[0125] In some embodiments, the third processing unit is also used to determine a first generation value corresponding to the first optical flow data, and multiple second generation values ​​corresponding to multiple second optical flow data; determine a cost fitting equation based on the first generation value, multiple second generation values, the first optical flow data, and multiple second optical flow data, and the cost fitting equation is used to indicate the correspondence between multiple optical flow data and cost values ​​of the first pixel point; determine a correction value of the first optical flow data based on the cost fitting equation.

[0126] In some embodiments, the fourth processing unit is also used to determine a coefficient matrix based on the cost fitting equation; determine a first correction value corresponding to the first optical flow data in the first direction and a second correction value corresponding to the second direction based on the coefficient matrix; use the first correction value to correct the first optical flow value of the first optical flow data in the first direction, and use the second correction value to correct the second optical flow value of the first optical flow data in the second direction; determine the target optical flow data based on the corrected first optical flow value and the corrected second optical flow value.

[0127] In some embodiments, the fourth processing unit is further used to determine that the first optical flow data is the target optical flow data when the first correction value or the second correction value cannot be determined based on the coefficient matrix; and to determine that the first optical flow data is the target optical flow data when the absolute value of the first correction value is greater than or equal to the first value, or the absolute value of the second correction value is greater than or equal to the first value.

[0128] In summary, the image processing device 500 can determine, based on the first optical flow data of the first pixel point, a plurality of second optical flow data that have an offset not equal to zero with the first optical flow data in the first direction and / or the second direction, and determine a correction value based on the first optical flow data and the plurality of second optical flow data, and then use the correction value to correct the first optical flow data, thereby optimizing the first optical flow data and calculating sub-pixel optical flow data, thereby improving the accuracy of the optical flow data.

[0129] In the embodiments provided above, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the various functions of the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the aforementioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the aforementioned functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0130] Figure 6 FIG6 is a block diagram of an electronic device 600 for implementing the above method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0131] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0132] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 602 may include one or more modules to facilitate interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate interaction between the multimedia component 608 and the processing component 602.

[0133] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0134] The power supply assembly 606 provides power to the various components of the electronic device 600. The power supply assembly 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0135] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0136] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 604 or transmitted via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0137] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0138] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600. The sensor assembly 614 can also detect changes in the position of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and temperature changes of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0139] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0140] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0141] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by the processor 620 of the electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0142] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.

[0143] Figure 7 FIG. 7 is a schematic diagram showing a structure of a chip 700 for implementing the above method according to an exemplary embodiment. Figure 7 The chip 700 includes a communication interface 701 and at least one processor 702. The communication interface 701 is used to receive signals input into the chip 700 or signals output from the above chip 700. The processor 702 communicates with the communication interface 701 and implements the method described in the above embodiments of the present disclosure through logic circuits or executing code instructions.

[0144] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0145] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in at least one embodiment or example.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0147] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection having at least one wire (control method), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0148] It should be understood that various parts of the embodiments of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0150] Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as standalone products, they may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0151] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Performing optical flow calculation on a first image and a second image acquired by the same image acquisition module to determine first optical flow data corresponding to a first pixel in the second image, where the first image is a frame image preceding the second image; Determining a plurality of second optical flow data according to the coordinates of the first optical flow data, wherein the coordinates of at least one of the plurality of second optical flow data are offset from the coordinates of the first optical flow data in both a first direction and a second direction and are not equal to zero; determining a correction value of the first optical flow data according to the first optical flow data and the plurality of second optical flow data; The first optical flow data is corrected using the correction value to obtain target optical flow data corresponding to the first pixel point.

2. The method according to claim 1, characterized in that Determining a plurality of second optical flow data according to the first optical flow data includes: Determining a maximum optical flow amount in the first direction and a maximum optical flow amount in the second direction; determining at least one offset of the first optical flow data in the first direction and / or the second direction according to the maximum optical flow amount in the first direction and the maximum optical flow amount in the second direction; The plurality of second optical flow data are determined according to the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

3. The method according to claim 2, characterized in that The determining the plurality of second optical flow data according to the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction comprises: Determine the maximum optical flow number in the first direction as N, and the maximum optical flow number in the second direction as M; offsetting the coordinates of the first optical flow data along the first direction according to the N-1 offsets of the first optical flow data in the first direction to obtain N-1 second optical flow data; offsetting the coordinates of the first optical flow data along the second direction according to the M-1 offsets of the first optical flow data in the second direction to obtain M-1 second optical flow data; According to the N-1 offsets of the first optical flow data in the first direction and the M-1 offsets in the second direction, the coordinates of the first optical flow data are offset along the first direction and the second direction to obtain N×MN-M+1 second optical flow data.

4. The method according to claim 1, wherein The determining, based on the first optical flow data and the plurality of second optical flow data, a correction value of the first optical flow data includes: Determining a first generation value corresponding to the first optical flow data and a plurality of second generation values ​​corresponding to the plurality of second optical flow data; Determining a cost fitting equation based on the first generation value, the multiple second generation values, the first optical flow data, and the multiple second optical flow data, where the cost fitting equation is used to indicate a correspondence between the multiple optical flow data of the first pixel and the cost value; A correction value of the first optical flow data is determined according to the cost fitting equation.

5. The method according to claim 4, characterized in that The correcting the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel point includes: Determine a coefficient matrix according to the cost fitting equation; Determining, according to the coefficient matrix, a first correction value corresponding to the first optical flow data in the first direction and a second correction value corresponding to the second direction; Correcting a first optical flow value of the first optical flow data in the first direction using the first correction value, and correcting a second optical flow value of the first optical flow data in the second direction using the second correction value; The target optical flow data is determined according to the corrected first optical flow value and the corrected second optical flow value.

6. The method according to claim 5, characterized in that The method further comprises: When the first correction value or the second correction value cannot be determined according to the coefficient matrix, determining the first optical flow data as the target optical flow data; When the absolute value of the first correction value is greater than or equal to a first value, or the absolute value of the second correction value is greater than or equal to the first value, the first optical flow data is determined to be the target optical flow data.

7. An image processing device, comprising: a first processing unit, configured to perform optical flow calculation on a first image and a second image acquired by the same image acquisition module, and determine first optical flow data corresponding to a first pixel in the second image, where the first image is a frame image preceding the second image; a second processing unit, configured to determine a plurality of second optical flow data according to the coordinates of the first optical flow data, wherein the coordinates of at least one of the plurality of second optical flow data are offset from the coordinates of the first optical flow data in both a first direction and a second direction and are not equal to zero; a third processing unit, configured to determine a correction value of the first optical flow data based on the first optical flow data and the plurality of second optical flow data; The fourth processing unit is configured to correct the first optical flow data using the correction value to obtain target optical flow data corresponding to the first pixel point.

8. The device according to claim 7, characterized in that The second processing unit is further configured to: Determining a maximum optical flow amount in the first direction and a maximum optical flow amount in the second direction; determining at least one offset of the first optical flow data in the first direction and / or the second direction according to the maximum optical flow amount in the first direction and the maximum optical flow amount in the second direction; The plurality of second optical flow data are determined according to the coordinates of the first optical flow data and at least one offset of the first optical flow data in the first direction and / or the second direction.

9. The device according to claim 7, characterized in that The third processing unit is further configured to: Determining a first generation value corresponding to the first optical flow data and a plurality of second generation values ​​corresponding to the plurality of second optical flow data; Determining a cost fitting equation based on the first generation value, the multiple second generation values, the first optical flow data, and the multiple second optical flow data, where the cost fitting equation is used to indicate a correspondence between the multiple optical flow data of the first pixel and the cost value; A correction value of the first optical flow data is determined according to the cost fitting equation.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

12. A chip, characterized in that: The method comprises at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method as described in any one of claims 1 to 6 through a logic circuit or executing code instructions.

Images