Image processing method and device and electronic equipment

Abstract

The invention relates to an image processing method and device and electronic equipment, and relates to the technical field of image processing. The method specifically comprises the following steps: carrying out optical flow calculation on a first image and a second image collected by the same camera at adjacent moments to obtain first optical flow data of each pixel on a pixel level; acquiring optical flow correction data of each pixel according to the first image, the second image and the first optical flow data; and for each pixel, obtaining second optical flow data of the pixel on a sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel. Therefore, the precision of the optical flow data is improved by acquiring the optical flow data of the pixel on the sub-pixel level, and meanwhile, the calculation amount is small and the implementation is easy when the optical flow data of the pixel on the sub-pixel level is acquired.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, and electronic device. Background Art

[0002] In the prior art, the Semi-Global Matching (SGM) algorithm can be used to calculate pixel-level dense optical flow data on two images. However, only pixel-level dense optical flow data can be obtained through the semi-global matching algorithm, and sub-pixel-level dense optical flow data cannot be obtained. Therefore, how to obtain sub-pixel-level dense optical flow data has become an urgent problem to be solved. Summary of the Invention

[0003] The present disclosure provides an image processing method, apparatus, and electronic device. Thus, the present disclosure improves the accuracy of optical flow data by obtaining the optical flow data of pixels at the sub-pixel level. At the same time, the calculation amount is small and it is easy to implement when obtaining the optical flow data of pixels at the sub-pixel level.

[0004] The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided. The method includes: performing optical flow calculation on a first image and a second image collected at adjacent moments by the same camera to obtain first optical flow data of each pixel at the pixel level; obtaining optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data; and for each pixel, obtaining second optical flow data of the pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel.

[0006] According to an embodiment of the present disclosure, the obtaining optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data includes: for each pixel, determining a second position of the pixel in the second image after optical flow based on the first optical flow data of the pixel and a first position of the pixel in the first image; and determining the optical flow correction data of the pixel according to the first position and the second position, where the optical flow correction data includes optical flow correction amounts in the first image direction and the second image direction.

[0007] According to an embodiment of the present disclosure, determining the optical flow correction data of the pixel based on the first position and the second position includes: for any image direction, determining a plurality of offset positions of the second image corresponding to the pixel in the any image direction based on the second position; obtaining cost values of the first position and each of the offset positions in the any image direction; and determining an optical flow correction amount of the pixel in the any image direction according to the cost values of each of the offset positions.

[0008] According to an embodiment of the present disclosure, determining a plurality of offset positions of the second image corresponding to the pixel in the any image direction based on the second position includes: determining a plurality of position offsets of the pixel in the any image direction; and determining the plurality of offset positions based on the second position and the plurality of position offsets.

[0009] According to an embodiment of the present disclosure, obtaining cost values of the first position and each of the offset positions in the any image direction includes: for each of the offset positions, determining a first window on the first image with the first position as a central position and a second window on the second image with the offset position as a central position; and performing cost calculation on the pixels covered in the first window and the pixels covered in the second window to obtain the cost value.

[0010] According to an embodiment of the present disclosure, determining the optical flow correction amount of the pixel in the any image direction according to the cost values of each of the offset positions includes: for the any image direction, performing linear fitting on the position offsets and the cost values of the plurality of offset positions in the any image direction to obtain a cost fitting curve; if the cost fitting curve is a concave curve, determining a position offset of a symmetry point of the cost fitting curve based on the cost values of the plurality of offset positions; and determining the optical flow correction amount of the pixel in the any image direction according to the position offset of the symmetry point.

[0011] According to an embodiment of the present disclosure, determining the optical flow correction amount of the pixel in the any image direction according to the position offset of the symmetry point includes: determining whether an absolute value of the position offset of the symmetry point is greater than a set threshold; if the absolute value is greater than the set threshold, determining the optical flow correction amount of the pixel in the any image direction as a set value; and if the absolute value is less than or equal to the set threshold, determining the position offset of the symmetry point as the optical flow correction amount of the pixel in the any image direction.

[0012] According to an embodiment of the present disclosure, the first optical flow data includes first optical flow values in a first image direction and a second image direction. Wherein, obtaining the second optical flow data of the pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel includes: for any one of the image directions, adding the first optical flow value of the pixel in any one of the image directions and the optical flow correction amount in any one of the image directions to obtain the second optical flow data of the pixel at the sub-pixel level in any one of the image directions. Wherein, the second optical flow data includes second optical flow values at the sub-pixel level in the first image direction and the second image direction.

[0013] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, the apparatus includes: a first acquisition module, configured to perform optical flow calculation on a first image and a second image acquired by the same camera at adjacent moments, and obtain first optical flow data of each pixel at the pixel level; a second acquisition module, configured to obtain optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data; a third acquisition module, configured to obtain second optical flow data of each pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel.

[0014] According to an embodiment of the present disclosure, the second acquisition module is further configured to: for each pixel, determine a second position of the pixel in the second image after optical flow based on the first optical flow data of the pixel and the first position of the pixel in the first image; determine the optical flow correction data of the pixel according to the first position and the second position.

[0015] According to an embodiment of the present disclosure, the second acquisition module is further configured to: for any one of the image directions, determine a plurality of offset positions of the second image corresponding to the pixel in any one of the image directions based on the second position; obtain cost values of the first position and each of the offset positions in any one of the image directions; determine the optical flow correction amount of the pixel in any one of the image directions according to the cost value of each of the offset positions. Wherein, the optical flow correction data includes optical flow correction amounts in the first image direction and the second image direction.

[0016] According to an embodiment of the present disclosure, the second acquisition module is further configured to: determine a plurality of position offsets of the pixel in any one of the image directions; determine the plurality of offset positions based on the second position and the plurality of position offsets.

[0017] According to an embodiment of the present disclosure, the second acquisition module is further configured to: for each of the offset positions, determine a first window on the first image with the first position as the center position, and determine a second window on the second image with the offset position as the center position; calculate a cost for the pixels covered in the first window and the pixels covered in the second window to obtain the cost value.

[0018] According to an embodiment of the present disclosure, the second acquisition module is further configured to: for any one of the image directions, perform a linear fit on the position offsets and cost values of a plurality of offset positions in the any one of the image directions to obtain a cost fitting curve; if the cost fitting curve is a concave curve, determine the position offset of the symmetry point of the cost fitting curve based on the cost values of the plurality of offset positions; according to the position offset of the symmetry point, determine the optical flow correction amount of the pixel in the any one of the image directions.

[0019] According to an embodiment of the present disclosure, the second acquisition module is further configured to: determine whether the absolute value of the position offset of the symmetry point is greater than a set threshold; if the absolute value is greater than the set threshold, determine the optical flow correction amount of the pixel in the any one of the image directions as a set value; if the absolute value is less than or equal to the set threshold, determine the position offset of the symmetry point as the optical flow correction amount of the pixel in the any one of the image directions.

[0020] According to an embodiment of the present disclosure, the apparatus is further configured to: if the cost fitting curve is a convex curve, determine the optical flow correction amount of the pixel in the any one of the image directions as a set value.

[0021] According to an embodiment of the present disclosure, the second acquisition module is further configured to: for the any one of the image directions, add the first optical flow value of the pixel in the any one of the image directions and the optical flow correction amount in the any one of the image directions to obtain the second optical flow data at the sub-pixel level of the pixel in the any one of the image directions, where the second optical flow data includes the second optical flow values at the sub-pixel levels of the first image direction and the second image direction.

[0022] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to execute the instructions to implement the image processing method provided in the embodiment of the first aspect of the present disclosure.

[0023] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the image processing method provided in the embodiment of the first aspect of the present disclosure.

[0024] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the image processing method provided in the first aspect of the present disclosure.

[0025] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0026] An image processing method according to an embodiment of the present disclosure calculates an optical flow for a first image and a second image captured at adjacent times by the same camera to obtain first optical flow data for each pixel at the pixel level; based on the first image, the second image, and the first optical flow data, optical flow correction data for each pixel is obtained; for each pixel, based on the first optical flow data of the pixel and the optical flow correction data of the pixel, second optical flow data for the pixel at the sub-pixel level is obtained. Thus, based on the optical flow data and the optical flow correction data at the pixel level, the present disclosure can obtain optical flow data for pixels at the sub-pixel level, improving the accuracy of the optical flow data. At the same time, when obtaining the optical flow data for pixels at the sub-pixel level, the amount of calculation is small and it is easy to implement.

[0027] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure and do not constitute an improper limitation of the present disclosure.

[0029] Figure 1 is a flowchart showing an image processing method according to an exemplary embodiment.

[0030] Figure 2 is a flowchart showing another image processing method according to an exemplary embodiment.

[0031] Figure 3 is a flowchart showing another image processing method according to an exemplary embodiment.

[0032] Figure 4 is a flowchart showing another image processing method according to an exemplary embodiment.

[0033] Figure 5 is a block diagram showing an image processing apparatus according to an exemplary embodiment.

[0034] Figure 6 is a block diagram showing an electronic device according to an exemplary embodiment. Detailed implementation manners

[0035] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] Figure 1 It is a schematic flowchart of an image processing method provided for an embodiment of the present disclosure.

[0038] As Figure 1 shown, the image processing method includes the following steps:

[0039] S101, perform optical flow calculation on the first image and the second image collected by the same camera at adjacent moments to obtain first optical flow data of each pixel at the pixel level.

[0040] Among them, optical flow data can be divided into sparse optical flow data and dense optical flow data. Sparse optical flow data is to select a small number of key points in the image and obtain sparse optical flow data by calculating the motion vectors of the key points. Dense optical flow data is to calculate the motion vector of each pixel point at each pixel point in the image to obtain dense optical flow data.

[0041] Optionally, the first optical flow data may be dense optical flow data.

[0042] It should be noted that the first optical flow data includes first optical flow values in the first image direction and the second image direction. The first image direction may be the horizontal x direction of the image, and the second image direction may be the vertical y direction of the image.

[0043] It should be noted that the first image and the second image at adjacent moments can be collected by the same camera. After obtaining the first image and the second image, optical flow calculation can be performed on the first image and the second image to obtain first optical flow data of each pixel at the pixel level.

[0044] It should be noted that the present disclosure does not limit the specific manner of performing optical flow calculation on the first image and the second image to obtain first optical flow data of each pixel at the pixel level, and can be selected according to actual situations.

[0045] Optionally, a semi-global matching algorithm (SGM for short) can be adopted, and the first optical flow values u and v in the first image direction and the second image direction, that is, the first optical flow data is (u, v).

[0046] S102. Obtain the optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data.

[0047] In the embodiments of the present disclosure, after obtaining the first optical flow data, for each pixel, based on the first optical flow data of the pixel and the first position of the pixel on the first image, determine the second position of the pixel on the second image after the optical flow, and according to the first position and the second position, determine the optical flow correction data of the pixel.

[0048] Optionally, for any image direction, based on the second position, determine multiple offset positions of the second image corresponding to the pixel in any image direction, obtain the cost values of the first position and each offset position in any image direction, and according to the cost values of each offset position, determine the optical flow correction amount of the pixel in any image direction.

[0049] It should be noted that the optical flow correction data includes the optical flow correction amounts m and n in the first image direction and the second image direction, that is, the optical flow correction data is (m, n).

[0050] S103. For each pixel, based on the first optical flow data of the pixel and the optical flow correction data of the pixel, obtain the second optical flow data of the pixel at the sub-pixel level.

[0051] Among them, the second optical flow data includes the second optical flow values at the sub-pixel level in the first image direction and the second image direction.

[0052] In the embodiments of the present disclosure, after obtaining the optical flow correction data, based on the first optical flow data of the pixel and the optical flow correction data, obtain the second optical flow data of the pixel at the sub-pixel level.

[0053] Optionally, for any image direction, the second optical flow data of the pixel at the sub-pixel level in any image direction can be obtained by adding the first optical flow value of the pixel in any image direction and the optical flow correction amount in any image direction.

[0054] For example, if the first optical flow data is (u, v) and the optical flow correction data is (m, n), then the second optical flow data is (u + m, v + n).

[0055] An image processing method according to an embodiment of the present disclosure calculates optical flow for a first image and a second image acquired at adjacent times by the same camera to obtain first optical flow data of each pixel at the pixel level; based on the first image, the second image, and the first optical flow data, obtains optical flow correction data of each pixel; for each pixel, based on the first optical flow data of the pixel and the optical flow correction data of the pixel, obtains second optical flow data of the pixel at the sub-pixel level. Thus, the present disclosure can obtain optical flow data of a pixel at the sub-pixel level based on optical flow data and optical flow correction data at the pixel level, improving the accuracy of the optical flow data. At the same time, when obtaining the optical flow data of a pixel at the sub-pixel level, the calculation amount is small and it is easy to implement.

[0056] Figure 2 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure. On the basis of the above embodiment, further combined with Figure 2 , the specific process of obtaining the optical flow correction data of each pixel based on the first image, the second image, and the first optical flow data is explained, including the following steps:

[0057] S201. For each pixel, based on the first optical flow data of the pixel and the first position of the pixel on the first image, determine the second position of the pixel on the second image after optical flow.

[0058] For example, if the first optical flow data of the pixel is (u, v) and the first position of the pixel on the first image is (x, y), then the second position of the pixel on the second image after optical flow is (x + u, y + v).

[0059] S202. Determine the optical flow correction data of the pixel according to the first position and the second position.

[0060] As a possible implementation, as Figure 3 shown, on the basis of the above steps, the specific process of determining the optical flow correction data of the pixel according to the first position and the second position in the above steps includes the following steps:

[0061] S301. For any image direction, based on the second position, determine multiple offset positions of the second image corresponding to the pixel in any image direction.

[0062] In an embodiment of the present application, for any image direction, multiple position offsets of the pixel in any image manner can be determined, and multiple offset positions are determined based on the second position and the multiple position offsets.

[0063] Optionally, the multiple position offsets can be: -1, 0, 1.

[0064] For example, for the first image direction, i.e., the horizontal x direction of the image, if the second position is (x + u, y + v), the multiple offset positions of the second image corresponding to the pixel are (x + u + 1, y + v), (x + u, y + v), and (x + u - 1, y + v).

[0065] For example, for the second image direction, i.e., the vertical y direction of the image, if the second position is (x + u, y + v), the multiple offset positions of the second image corresponding to the pixel are (x + u, y + v + 1), (x + u, y + v), and (x + u, y + v - 1).

[0066] S302, obtain the cost values of the first position and each offset position in any image direction.

[0067] In the embodiments of the present disclosure, for each offset position, a first window is determined on the first image with the first position as the center position, and a second window is determined on the second image with the offset position as the center position. The cost calculation is performed on the pixels covered in the first window and the pixels covered in the second window to obtain the cost value.

[0068] It should be noted that the present disclosure does not limit the specific manner of calculating the cost value by performing cost calculation on the pixels covered in the first window and the pixels covered in the second window, and can be selected according to the actual situation.

[0069] Optionally, the cost calculation can be performed on the pixels covered in the first window and the pixels covered in the second window by the way of the sum of absolute differences (SAD) to obtain the cost value.

[0070] Optionally, the cost calculation can be performed on the pixels covered in the first window and the pixels covered in the second window by the way of Census transform and Hamming distance to obtain the cost value.

[0071] For example, for the first image direction, i.e., the horizontal x direction of the image, if the second position is (x + u, y + v) and the offset positions are (x + u + 1, y + v), (x + u, y + v), and (x + u - 1, y + v), the cost value P of the second position (x + u, y + v) and the offset position (x + u + 1, y + v) in the first image direction can be obtained based on the SAD method 0 、the cost value P of the second position (x + u, y + v) and the offset position (x + u, y + v) in the first image direction 1 、the cost value P of the second position (x + u, y + v) and the offset position (x + u - 1, y + v) in the first image direction 2 .

[0072] For example, for the second image direction, i.e., the image vertical y direction, if the second position is (x + u, y + v), the offset positions are (x + u, y + v + 1), (x + u, y + v), (x + u, y + v - 1), and based on the SAD method, the cost value Q of the second position x + u, y + v and the offset position x + u, y + v + 1 in the second image direction can be obtained. 0 The cost value Q1 of the second position (x + u, y + v) and the offset position (x + u, y + v) in the second image direction, and the cost value Q2 of the second position x + u, y + v and the offset position x + u, y + v - 1 in the second image direction.

[0073] S303. Determine the optical flow correction amount of the pixel in any image direction according to the cost value of each offset position, where the optical flow correction data includes the optical flow correction amounts in the first image direction and the second image direction.

[0074] As a possible implementation, as Figure 4 shown, based on the above steps, the specific process of determining the optical flow correction amount of the pixel in any image direction according to the cost value of each offset position in the above steps includes the following steps:

[0075] S401. For any image direction, perform linear fitting on the position offsets and cost values of multiple offset positions in any image direction to obtain a cost fitting curve.

[0076] Optionally, the cost fitting curve can be a quadratic equation curve. Correspondingly, a quadratic equation y = a * x 2 + b * x + c can be constructed, where x is the offset, and a, b, c are fitting coefficients. The above quadratic equation can be fitted based on the cost value and the offset to obtain the cost fitting curve.

[0077] For example, for the first image direction, i.e., the image horizontal x direction, the offsets are -1, 0, 1, and the corresponding cost values are P 0 、P 1 、P 2 ,y = a * x 2 + b * x + c can be fitted to obtain the cost fitting curve.

[0078] S402. If the cost fitting curve is a concave curve, determine the position offset of the symmetry point of the cost fitting curve based on the cost values of multiple offset positions.

[0079] It should be noted that if the cost fitting curve is a concave curve, that is, the fitting coefficient a in the above quadratic equation is greater than zero, the position offset of the symmetry point of the cost fitting curve determined based on the cost values of multiple offset positions is:

[0080] In the embodiments of the present disclosure, if the cost fitting curve is a convex curve, the optical flow correction amount of the pixel in any image direction is determined as a set value.

[0081] For example, if the cost fitting curve is a convex curve, that is, the fitting coefficient a in the above quadratic equation is less than zero, the optical flow correction amount of the pixel in the first image direction is a set value.

[0082] Optionally, the set value can be set to 0. If the cost fitting curve is a convex curve, the optical flow correction amount of the pixel in any image direction is determined to be 0.

[0083] S403. Determine the optical flow correction amount of the pixel in any image direction according to the position offset of the symmetric point.

[0084] In the embodiments of the present disclosure, after obtaining the position offset of the symmetric point, the optical flow correction amount of the pixel in any image direction can be determined according to the position offset of the symmetric point.

[0085] In the embodiments of the present disclosure, it can be determined whether the absolute value of the position offset of the symmetric point is greater than a set threshold, and according to the judgment result, the optical flow correction amount of the pixel in any image direction is determined.

[0086] It should be noted that the present disclosure does not limit the set threshold, and it can be set according to the actual situation. For example, the set threshold can be 1.

[0087] Optionally, if the absolute value of the position offset is greater than the set threshold, the optical flow correction amount of the pixel in any image direction is determined as a set value.

[0088] For example, for the first image direction, that is, the horizontal x direction of the image, the absolute value of the position offset is When then the optical flow correction amount of the pixel in the first image direction is determined to be 0.

[0089] Optionally, if the absolute value of the position offset is less than or equal to the set threshold, the position offset of the symmetric point is determined as the optical flow correction amount of the pixel in the first image direction.

[0090] For example, for the first image direction, that is, the horizontal x direction of the image, the absolute value of the position offset is When then the optical flow correction amount of the pixel in the first image direction is determined to be

[0091] It should be noted that the process of obtaining the optical flow correction amount in the second image direction is the same as that of obtaining the optical flow correction amount in the first image direction, and will not be elaborated here.

[0092] According to an image processing method of an embodiment of the present disclosure, for each pixel, based on the first optical flow data of the pixel and the first position of the pixel on the first image, determine the second position of the pixel on the second image after optical flow, and according to the first position and the second position, determine the optical flow correction data of the pixel. Based on the first optical flow data of the pixel and the optical flow correction data of the pixel, obtain the second optical flow data of the pixel at the sub-pixel level. Thus, the present disclosure improves the accuracy of the optical flow data by obtaining the optical flow data of the pixel at the sub-pixel level. At the same time, the calculation amount is small and easy to implement when obtaining the optical flow data of the pixel at the sub-pixel level, and the accuracy of image alignment and the accuracy of the parallax calculation result are improved.

[0093] Figure 5 It is a block diagram of an image processing device shown according to an exemplary embodiment.

[0094] As Figure 5 shown, the image processing device 1000 includes: a first acquisition module 110, a second acquisition module 120, and a third acquisition module 130.

[0095] The first acquisition module 110 is configured to perform optical flow calculation on the first image and the second image acquired by the same camera at adjacent moments, and obtain the first optical flow data of each pixel at the pixel level;

[0096] The second acquisition module 120 is configured to obtain the optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data;

[0097] The third acquisition module 130 is configured to, for each pixel, obtain the second optical flow data of the pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel.

[0098] Further, the second acquisition module 120 is further configured to: for each pixel, based on the first optical flow data of the pixel and the first position of the pixel on the first image, determine the second position of the pixel on the second image after optical flow; and according to the first position and the second position, determine the optical flow correction data of the pixel.

[0099] Further, the second acquisition module 120 is further configured to: for any image direction, based on the second position, determine a plurality of offset positions of the second image corresponding to the pixel in the any image direction; obtain the cost values of the first position and each of the offset positions in the any image direction; and according to the cost values of each of the offset positions, determine the optical flow correction amount of the pixel in the any image direction.

[0100] Further, the second obtaining module 120 is further configured to: determine a plurality of position offsets of the pixel in any of the image directions; and determine the plurality of offset positions based on the second position and the plurality of position offsets.

[0101] Further, the second obtaining module 120 is further configured to: for each of the offset positions, determine a first window on the first image with the first position as the center position, and determine a second window on the second image with the offset position as the center position; calculate a cost for the pixels covered in the first window and the pixels covered in the second window to obtain the cost value.

[0102] Further, the second obtaining module 120 is further configured to: for any of the image directions, perform linear fitting on the position offsets and cost values of the plurality of offset positions in any of the image directions to obtain a cost fitting curve; if the cost fitting curve is a concave curve, determine the position offset of the symmetry point of the cost fitting curve based on the cost values of the plurality of offset positions; and determine the optical flow correction amount of the pixel in any of the image directions according to the position offset of the symmetry point.

[0103] Further, the second obtaining module 120 is further configured to: determine whether the absolute value of the position offset of the symmetry point is greater than a set threshold; if the absolute value is greater than the set threshold, determine the optical flow correction amount of the pixel in any of the image directions as a set value; if the absolute value is less than or equal to the set threshold, determine the position offset of the symmetry point as the optical flow correction amount of the pixel in any of the image directions.

[0104] Further, the apparatus 1000 is further configured to: if the cost fitting curve is a convex curve, determine the optical flow correction amount of the pixel in any of the image directions as a set value.

[0105] Further, the second obtaining module 120 is further configured to: for any of the image directions, add the first optical flow value of the pixel in any of the image directions and the optical flow correction amount in any of the image directions to obtain the second optical flow data at the sub-pixel level of the pixel in any of the image directions, where the second optical flow data includes the second optical flow values at the sub-pixel levels of the first image direction and the second image direction.

[0106] An image processing apparatus according to an embodiment of the present disclosure calculates an optical flow between a first image and a second image acquired at adjacent times by the same camera to obtain first optical flow data of each pixel at a pixel level; obtains optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data; and for each pixel, obtains second optical flow data of the pixel at a sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel. Thus, based on the optical flow data and the optical flow correction data at the pixel level, the present disclosure can obtain optical flow data of a pixel at a sub-pixel level, improving the accuracy of the optical flow data. At the same time, when obtaining the optical flow data of a pixel at a sub-pixel level, the calculation amount is small and it is easy to implement.

[0107] To implement the above embodiment, the present disclosure also provides an electronic device, as Figure 6 shown, the electronic device 2000 includes: a processor 201; one or more memories 202 for storing executable instructions of the processor 201; wherein the processor 201 is configured to execute the image processing method described in the above embodiment. The processor 201 and the memory 202 are connected through a communication bus.

[0108] To implement the above embodiment, the present disclosure also provides a computer-readable storage medium including instructions, such as the memory 202 including instructions, and the above instructions can be executed by the processor 201 of the device 1000 to complete the above method. Optionally, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0109] To implement the above embodiment, the present disclosure also provides a computer program product, including a computer program, characterized in that the computer program, when executed by a processor, implements the image processing method described in the above embodiment.

[0110] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, the method includes: Performing optical flow calculation on the first image and the second image collected by the same camera at adjacent times to obtain first optical flow data of each pixel at the pixel level; According to the first image, the second image, and the first optical flow data, obtaining optical flow correction data of each pixel; For each pixel, based on the first optical flow data of the pixel and the optical flow correction data of the pixel, obtaining second optical flow data of the pixel at the sub-pixel level.

2. The method according to claim 1, characterized in that, the obtaining of the optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data includes: For each pixel, based on the first optical flow data of the pixel and the first position of the pixel on the first image, determining the second position of the pixel on the second image after optical flow; According to the first position and the second position, determining the optical flow correction data of the pixel.

3. The method according to claim 2, characterized in that, the determining of the optical flow correction data of the pixel according to the first position and the second position includes: For any image direction, based on the second position, determining a plurality of offset positions of the second image corresponding to the pixel in the any image direction; Obtaining the cost values of the first position and each of the offset positions in the any image direction; According to the cost values of each of the offset positions, determining the optical flow correction amount of the pixel in the any image direction, wherein the optical flow correction data includes the optical flow correction amounts in the first image direction and the second image direction.

4. The method according to claim 3, characterized in that, the determining of a plurality of offset positions of the second image corresponding to the pixel in the any image direction based on the second position includes: Determining a plurality of position offsets of the pixel in the any image direction; Based on the second position and the plurality of position offsets, determining the plurality of offset positions.

5. The method according to claim 3, characterized in that, the obtaining of the cost values of the first position and each of the offset positions in the any image direction includes: For each of the offset positions, determining a first window on the first image with the first position as the center position, and determining a second window on the second image with the offset position as the center position; Performing cost calculation on the pixels covered in the first window and the pixels covered in the second window to obtain the cost value.

6. The method according to claim 3, characterized in that, the determining of the optical flow correction amount of the pixel in the any image direction according to the cost values of each of the offset positions includes: For the any image direction, performing linear fitting on the position offsets and cost values of the plurality of offset positions in the any image direction to obtain a cost fitting curve; If the cost fitting curve is a concave curve, determining the position offset of the symmetry point of the cost fitting curve based on the cost values of the plurality of offset positions; Determine the optical flow correction amount of the pixel in any image direction according to the position offset of the symmetric point.

7. The method according to claim 6, wherein, the determining the optical flow correction amount of the pixel in any image direction according to the position offset of the symmetric point includes: judging whether the absolute value of the position offset of the symmetric point is greater than a set threshold; if the absolute value is greater than the set threshold, determining the optical flow correction amount of the pixel in any image direction as a set value; if the absolute value is less than or equal to the set threshold, determining the position offset of the symmetric point as the optical flow correction amount of the pixel in any image direction.

8. The method according to claim 6, wherein, the method further includes: if the cost fitting curve is a convex curve, determining the optical flow correction amount of the pixel in any image direction as a set value.

9. The method according to any one of claims 3-8, wherein, the first optical flow data includes first optical flow values in a first image direction and a second image direction, and wherein obtaining the second optical flow data of the pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel includes: for any one of the image directions, adding the first optical flow value of the pixel in any one of the image directions and the optical flow correction amount in any one of the image directions to obtain the second optical flow data of the pixel at the sub-pixel level in any one of the image directions, wherein the second optical flow data includes second optical flow values at the sub-pixel level in the first image direction and the second image direction.

10. An image processing apparatus, wherein, the apparatus includes: a first acquisition module, configured to perform optical flow calculation on a first image and a second image acquired by the same camera at adjacent moments, and acquire first optical flow data of each pixel at the pixel level; a second acquisition module, configured to acquire optical flow correction data of each pixel according to the first image, the second image, and the first optical flow data; a third acquisition module, configured to obtain, for each pixel, second optical flow data of the pixel at the sub-pixel level based on the first optical flow data of the pixel and the optical flow correction data of the pixel.

11. An electronic device, wherein, it includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the image processing method according to any one of claims 1-9 is implemented.

12. A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, the image processing method according to any one of claims 1-9 is implemented.

Images