Dense optical flow generation method and apparatus, and device, storage medium and program product

Abstract

Disclosed in the embodiments of the present application are a dense optical flow generation method and apparatus, and a device and a storage medium. The method comprises: performing block matching on the current image frame and the next image frame, in order to obtain a first motion estimation vector corresponding to each pixel point to be processed in the current image frame, wherein the first motion estimation vector corresponding to said pixel point is determined on the basis of a block matching result corresponding to a unit image block in which said pixel point is located; on the basis of the matching relationship between a local image in which each said pixel point is located in the current image frame and a local image in which a corresponding matching pixel point is located in the next image frame, fitting a second motion estimation vector corresponding to each said pixel point, wherein the matching pixel point is determined on the basis of said pixel point and the corresponding first motion estimation vector; and on the basis of the first motion estimation vector and the second motion estimation vector corresponding to each said pixel point, generating dense optical flow information.

Description

Dense optical flow generation method, device, equipment, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on and claims the priority of Chinese patent application with application number 202311326417.2, application date October 12, 2023, and application name “Dense Optical Flow Generation Method, Device, Equipment and Storage Medium”. The entire content of this Chinese patent application is hereby introduced into this disclosure as a reference. Technical Field

[0003] The present application relates to, but is not limited to, the field of image processing technology, and in particular to a dense optical flow generation method, apparatus, device, storage medium, and program product. Background Art

[0004] In the field of computer vision, optical flow describes the motion trajectory of pixels in an image or the correspondence between pixels in a pair of images. Optical flow generally includes sparse optical flow and dense optical flow. Sparse optical flow generally describes salient feature points, while dense optical flow describes all pixels in an image. In image processing tasks such as action recognition and motion prediction, optical flow plays a vital role as a motion feature. Therefore, accurately estimating optical flow is particularly important in the field of computer vision. Traditional dense optical flow estimation methods often require a large number of iterations, resulting in long computation times and inability to achieve high computational speeds.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present application at least provide a dense optical flow generation method, apparatus, device, storage medium, and program product.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] On the one hand, an embodiment of the present application provides a dense optical flow generation method, which includes: performing block matching on a current image frame and a next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; the first motion estimation vector corresponding to the pixel to be processed is determined by the block matching result corresponding to the unit image block where the pixel to be processed is located; based on the matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame, fitting the second motion estimation vector corresponding to each pixel to be processed; the matching pixel is determined based on the pixel to be processed and the corresponding first motion estimation vector; based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed, generating dense optical flow information.

[0009] In some embodiments, the first motion estimation vector corresponding to the pixel to be processed is the target block motion vector of the unit image block where the pixel to be processed is located; the block matching of the current image frame and the next image frame to obtain the first motion estimation vector corresponding to each pixel to be processed in the current image frame includes: for each preset image block in the current image frame, determining the image block pixel difference of the preset image block at different division scales; based on the image block pixel difference of the preset image block at different division scales, determining the target block motion vector of each unit image block in the preset image block.

[0010] In some embodiments, the different division scales include at least a first division scale corresponding to the preset image block, and determining the target block motion vector of each unit image block in the preset image block based on the image block pixel difference of the preset image block at different division scales includes: in response to the image block pixel difference at the first division scale being greater than or equal to the image block pixel difference at any other division scale, determining, for each unit image block, the target block motion vector of the unit image block based on the block motion vector corresponding to the unit image block at each division scale.

[0011] In some embodiments, the different division scales include at least a first division scale corresponding to the preset image block, and determining the target block motion vector of each unit image block in the preset image block based on the image block pixel difference of the preset image block at different division scales includes: in response to the image block pixel difference at the first division scale being smaller than the image block pixel difference at each other division scale, determining the block motion vector corresponding to the preset image block as the target block motion vector of each unit image block in the preset image block.

[0012] In some embodiments, determining the image block pixel difference of the preset image block at different division scales includes: for each division scale, determining the image block pixel difference between a first sub-image block and a corresponding second sub-image block at the division scale; the first sub-image block is obtained by dividing the preset image block based on the division scale; the second sub-image block is the image block in the next image frame having the smallest image block pixel difference with the first sub-image block; and taking the sum of the image block pixel differences corresponding to the first sub-image block at the division scale as the image block pixel difference of the preset image block at the division scale.

[0013] In some embodiments, determining the target block motion vector of the unit image block based on the block motion vector corresponding to the unit image block at each of the division scales includes: generating an image block pixel difference corresponding to the unit image block at each of the division scales based on the block motion vector corresponding to the unit image block at each of the division scales; and determining the block motion vector corresponding to the minimum image block pixel difference as the target block motion vector of the unit image block.

[0014] In some embodiments, the method further comprises:

[0015] For each first sub-image block at each of the division scales, a search area corresponding to the first sub-image block is determined in the next image frame; the first sub-image block is obtained by dividing the preset image block based on the division scale; within the search area corresponding to the first sub-image block, a sliding window corresponding to the size of the first sub-image block is used to intercept the selected image blocks, and the image block pixel difference between each of the selected image blocks and the first sub-image block is calculated; the selected image block corresponding to the smallest image block pixel difference is determined as the second sub-image block corresponding to the first sub-image block; based on the first sub-image block and the corresponding second sub-image block, a block motion vector corresponding to the first sub-image block at the division scale is generated; wherein the block motion vector corresponding to the unit image block within the first sub-image block at the division scale is the same as the block motion vector corresponding to the first sub-image block.

[0016] In some embodiments, the fitting of the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame includes: for each pixel to be processed, obtaining multiple neighborhood pixel points corresponding to the pixel to be processed in the current image frame, and obtaining multiple neighborhood pixel points corresponding to the matching pixel in the next image frame; fitting the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between the multiple neighborhood pixel points corresponding to the pixel to be processed and the multiple neighborhood pixel points corresponding to the matching pixel.

[0017] In some embodiments, fitting the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between multiple neighboring pixel points corresponding to the pixel to be processed and multiple neighboring pixel points corresponding to the matching pixel point includes: constructing a first matrix function based on the multiple neighboring pixel points corresponding to the pixel to be processed; the first matrix function includes a first pixel function of each neighboring pixel point corresponding to the pixel to be processed, and the first pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighboring pixel point in the current image frame and the grayscale value of the neighboring pixel point; constructing a second matrix function based on the multiple neighboring pixel points corresponding to the neighboring pixel point; the second matrix function includes a second pixel function of each neighboring pixel point corresponding to the neighboring pixel point, and the second pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighboring pixel point in the next image frame and the grayscale value of the neighboring pixel point; generating the second motion estimation vector corresponding to the pixel to be processed based on the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function.

[0018] In some embodiments, the quadratic polynomial coefficients of the pixel function include matrix coefficients and vector coefficients; the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function is used to generate the second motion estimation vector corresponding to the pixel point to be processed, including: determining the mean matrix coefficient based on the first matrix coefficient corresponding to the first pixel function and the second matrix coefficient corresponding to the second pixel function; determining the difference vector coefficient based on the first vector coefficient corresponding to the first pixel function and the second vector coefficient corresponding to the second pixel function; and determining the product of the inverse matrix corresponding to the mean matrix coefficient, the preset coefficient and the difference vector coefficient as the second motion estimation vector corresponding to the pixel point to be processed.

[0019] In some embodiments, the dense optical flow information includes a target motion estimation vector corresponding to each of the pixel points to be processed; the generation of dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each of the pixel points to be processed includes: for each of the pixel points to be processed, based on a preset pixel difference calculation window, determining the first calculation window pixel difference corresponding to the pixel point to be processed under the first motion estimation vector and the second calculation window pixel difference corresponding to the pixel point to be processed under the second motion estimation vector; and taking the motion estimation vector corresponding to the minimum calculation window pixel difference between the first calculation window pixel difference and the second calculation window pixel difference as the target motion estimation vector corresponding to the pixel point to be processed.

[0020] On the other hand, an embodiment of the present application provides a dense optical flow generation device, comprising:

[0021] The first vector acquisition unit is configured to perform block matching on a current image frame and a next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; the first motion estimation vector corresponding to the pixel to be processed is determined by a block matching result corresponding to a unit image block where the pixel to be processed is located;

[0022] The second vector acquisition part is configured to fit a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a partial image of each pixel to be processed in a current image frame and a partial image of a corresponding matching pixel in a next image frame; the matching pixel is determined based on the pixel to be processed and the corresponding first motion estimation vector;

[0023] The dense optical flow generating part is configured to generate dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed.

[0024] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements some or all of the steps in the above method.

[0025] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0026] On the other hand, an embodiment of the present application provides a computer program product, including computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0027] In the embodiment of the present application, the first motion estimation vector of the pixel to be processed at the image block level and the second motion estimation vector of the pixel to be processed at the pixel level are obtained, and the final dense optical flow information is generated based on the first motion estimation vector and the second motion estimation vector. In this way, the obtained dense optical flow information not only focuses on the global information of the image but also the local information of the image, thereby improving the accuracy of the dense optical flow information. At the same time, by using the first motion estimation vector at the image block level to determine the preliminary direction during the pixel-level motion estimation process, compared with the dense optical flow calculation scheme of the related art, because the entire image information of the image frame does not need to be considered, multiple iterations are eliminated, thereby reducing the optical flow calculation time.

[0028] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0030] FIG1 is a schematic diagram of a first implementation flow of a dense optical flow generation method provided in an embodiment of the present application;

[0031] FIG2 is a second schematic diagram of an implementation flow of a dense optical flow generation method provided in an embodiment of the present application;

[0032] FIG3A is a third schematic diagram of an implementation flow of a dense optical flow generation method provided in an embodiment of the present application;

[0033] FIG3B is a schematic diagram showing the relationship between neighboring pixels in a current image frame and a next image frame provided by an embodiment of the present application;

[0034] FIG4 is a fourth schematic diagram of an implementation flow of a dense optical flow generation method provided in an embodiment of the present application;

[0035] FIG5 is a schematic diagram of data flow of a dense optical flow calculation system provided in an embodiment of the present application;

[0036] FIG6 is a schematic diagram of motion estimation of a unit image block provided by an embodiment of the present application;

[0037] FIG7 is a flowchart of motion estimation of a unit image block in a refined division state provided by an embodiment of the present application;

[0038] FIG8 is a schematic diagram of a motion estimation vector of a dense optical flow provided in an embodiment of the present application;

[0039] FIG9 is a schematic diagram of a neighborhood point provided in an embodiment of the present application;

[0040] FIG10 is a schematic diagram of the structure of a dense optical flow generating device provided in an embodiment of the present application;

[0041] FIG11 is a schematic diagram of a hardware entity of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0043] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or sequence of "first / second / third" may be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0045] The present invention provides a method for generating dense optical flow, which can be executed by a processor of a computer device. The computer device may include a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities.

[0046] [Corrected 30.10.2024 according to Rule 91] FIG1 is a schematic diagram of a first implementation flow of a dense optical flow generation method provided in an embodiment of the present application. As shown in FIG1 , the method includes the following steps S101 to S103:

[0047] Step S101: Block matching is performed on the current image frame and the next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; the first motion estimation vector corresponding to the pixel to be processed is determined by the block matching result corresponding to the unit image block where the pixel to be processed is located.

[0048] In some embodiments, the above-mentioned block matching method takes the unit image block as the matching target, and generates a block matching result corresponding to each unit image block in the current image frame based on the block matching algorithm, that is, the target image block corresponding to each unit image block is determined in the next image frame, and then the first motion estimation vector corresponding to each unit image block is generated based on each unit image block and the corresponding target image block.

[0049] In an embodiment of the present application, the processor sets the first motion estimation vector of each pixel to be processed in the unit image block to the first motion estimation vector corresponding to the unit image block; that is, for all pixel points to be processed in the same unit image block, their corresponding first motion estimation vectors are the same.

[0050] It can be understood that the first motion estimation vector is a block-level motion estimation vector determined by taking into account texture information of the unit image block where the pixel point is located.

[0051] Step S102: Fitting a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a local image of each pixel to be processed in the current image frame and a local image of a corresponding matching pixel in the next image frame; the matching pixel is determined based on the pixel to be processed and the corresponding first motion estimation vector.

[0052] In some embodiments, for each of the pixel points to be processed, the position coordinates of the matching pixel point in the next image frame can be determined based on the position coordinates of the pixel point to be processed in the current image frame and the first motion estimation vector corresponding to the pixel point to be processed, thereby determining the matching pixel point.

[0053] In this embodiment of the present application, considering that the motion of objects in adjacent frames is relatively "small," that is, for a pixel to be processed, the image information of the local image in which the pixel is located is substantially the same as the local image of the matching pixel in the next image frame. Therefore, for each pixel to be processed, step S102 can fit a second motion estimation vector corresponding to the pixel to be processed based on the matching relationship between the local image of the pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame.

[0054] In some embodiments, the processor may fit the second motion estimation vector corresponding to the pixel to be processed based on an optical flow method, wherein the optical flow method may be any one of the following: Horn-Schunck optical flow method, Lucas-Kanade optical flow method, Farneback optical flow method, etc.

[0055] It can be understood that the second motion estimation vector is a pixel-level motion estimation vector determined by taking into account texture information of the local image where the pixel point is located.

[0056] Step S103: Generate dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed.

[0057] In an embodiment of the present application, for each pixel to be processed, the difference in image information corresponding to the first motion estimation vector and the difference in image information corresponding to the second motion estimation vector are determined between the current image frame and the next image frame, and the motion estimation vector with the smaller difference is used as the final motion estimation vector of the pixel to be processed, thereby obtaining the dense optical flow information of the entire current image frame and the next image frame.

[0058] In some embodiments, based on the position coordinates of the pixel point to be processed in the current image frame and the first motion estimation vector, the matching position coordinates can be determined in the next image frame, and the pixel value at the matching position coordinates can be obtained as the pixel value corresponding to the first motion estimation vector; based on the same method, the pixel value corresponding to the second motion estimation vector is determined, and the difference between the pixel value corresponding to the pixel point to be processed and the pixel value corresponding to the first motion estimation vector, and the difference between the pixel value corresponding to the pixel point to be processed and the pixel value corresponding to the second motion estimation vector are determined respectively; the motion estimation vector with the smaller difference is used as the final motion estimation vector of the pixel point to be processed.

[0059] In some embodiments, the matching position coordinates can be determined in the next image frame based on the position coordinates of the pixel to be processed in the current image frame and the first motion estimation vector, and the local image at the matching position coordinates can be obtained as the local image corresponding to the first motion estimation vector; based on the same method, the local image corresponding to the second motion estimation vector can be determined, and the difference between the local image corresponding to the pixel to be processed and the local image corresponding to the first motion estimation vector, and the difference between the local image corresponding to the pixel to be processed and the local image corresponding to the second motion estimation vector can be determined respectively; the motion estimation vector with the smaller difference can be used as the final motion estimation vector of the pixel to be processed.

[0060] In other embodiments, the first motion estimation vector and the second motion estimation vector can be directly weighted averaged, and the fused motion estimation vector can be used as the final motion estimation vector of the pixel to be processed, thereby obtaining dense optical flow information of the entire current image frame and the next image frame.

[0061] In the embodiment of the present application, the first motion estimation vector of the pixel to be processed at the image block level and the second motion estimation vector of the pixel to be processed at the pixel level are obtained, and the final dense optical flow information is generated based on the first motion estimation vector and the second motion estimation vector. In this way, the obtained dense optical flow information not only focuses on the global information of the image but also the local information of the image, thereby improving the accuracy of the dense optical flow information. At the same time, by using the first motion estimation vector at the image block level to determine the preliminary direction during the pixel-level motion estimation process, compared with the dense optical flow calculation scheme of the related art, because the entire image information of the image frame does not need to be considered, multiple iterations are eliminated, thereby reducing the optical flow calculation time.

[0062] FIG2 is a second optional flow diagram of a dense optical flow generation method provided in an embodiment of the present application, which can be executed by a processor of a computer device. Based on FIG1 , the first motion estimation vector corresponding to the pixel to be processed is the target block motion vector of the unit image block in which the pixel to be processed resides; S101 in FIG1 can be updated to S201 through S203, which will be explained in conjunction with the steps shown in FIG2 .

[0063] Step S201 : for each preset image block in the current image frame, determining the image block pixel difference of the preset image block at different division scales.

[0064] In some embodiments, the preset image block needs to be divided at different division scales, that is, the preset image block is divided at at least two division scales, so as to obtain at least one first sub-image block corresponding to each division scale.

[0065] The preset image block includes a plurality of the unit image blocks.

[0066] In an embodiment of the present application, step S101 may be performed by a video codec, and the preset image block is the smallest processing unit of the video codec. For example, the size of the preset image block may be set to 32*32, i.e., the preset image block is a square image block with a side length of 32 pixels. The current image frame may be divided into a plurality of preset image blocks, and the processor may execute step S201 for each of the plurality of preset image blocks.

[0067] In some embodiments, to obtain more refined motion estimation data, the preset image block can be further divided to obtain multiple unit image blocks corresponding to the preset image block. The current embodiment determines the target block motion vector corresponding to each unit image block and then uses the target block motion vector corresponding to the unit image block as the target block motion vector for each pixel to be processed in the unit image block.

[0068] It is understandable that the size of the unit image block is smaller than the size of the preset image block. In some embodiments, a preset image block may include 4 to the power of n unit image blocks, where n is an integer greater than or equal to 1. For example, when the size of the preset image block is 32*32, the size of the unit image block may be 16*16, in which case, a preset image block may include 4 unit image blocks; when the size of the preset image block is 32*32, the size of the unit image block may be 8×8, in which case, a preset image block may include 16 unit image blocks; when the size of the preset image block is 32*32, the size of the unit image block may be 4*4, in which case, a preset image block may include 64 unit image blocks, and so on.

[0069] In some embodiments, the different division scales include at least a first division scale and a second division scale, wherein the first sub-image block corresponding to the first division scale is the same size as the preset image block; and the first sub-image block corresponding to the second division scale is the same size as the unit image block.

[0070] In an embodiment of the present application, at least one third division scale is provided in addition to the first division scale and the second division scale. It can be understood that the size of the first sub-image block corresponding to the third division scale is smaller than the size of the first sub-image block corresponding to the first division scale. At the same time, the size of the first sub-image block corresponding to the third division scale is larger than the size of the first sub-image block corresponding to the second division scale.

[0071] For example, when the size of the preset image block is 32*32, the size of the first sub-image block corresponding to the first division scale is also 32*32, that is, the preset image block is directly used as the first sub-image block corresponding to the first division scale; accordingly, when the size of the unit image block is 8×8, the size of the first sub-image block corresponding to the second division scale is 8×8, that is, the preset image block is directly divided into 16 first sub-image blocks corresponding to the second division scale. It is understandable that the different division scales can also include a third division scale, and the size of the first sub-image block corresponding to the third division scale is 16*16, that is, the preset image block is directly divided into 4 first sub-image blocks corresponding to the third division scale.

[0072] In this embodiment, the pixel difference of the image block at the division scale is related to the motion estimation of each first sub-image block at the division scale.

[0073] It can be understood that, for each of the division scales, block matching can be performed on each of the first sub-image blocks under the division scale in the next frame image, and the minimum image block pixel difference corresponding to each first sub-image block in the block matching process is obtained, and the sum of the minimum image block pixel differences corresponding to all the first sub-image blocks under the division scale is used as the image block pixel difference of the preset image block at the division scale.

[0074] In some embodiments, the above-mentioned determination of the pixel difference of the preset image block at different division scales can be achieved through steps S2011 to S2012.

[0075] Step S2011 : for each of the division scales, respectively determine the image block pixel difference between the first sub-image block and the corresponding second sub-image block at the division scale.

[0076] The first sub-image block is obtained by dividing the preset image block based on the division scale; and the second sub-image block is an image block in the next image frame having the smallest image block pixel difference with the first sub-image block.

[0077] In an embodiment of the present application, for the first sub-image block and the corresponding second sub-image block, the difference in pixel values ​​is taken pixel by pixel, and the cumulative sum of the differences in pixel values ​​of each pixel is used as the image block pixel difference between the first sub-image block and the corresponding second sub-image block.

[0078] A second sub-image block matching the first sub-image block can be determined in the next image frame using a block matching method. In some embodiments, the second sub-image block is an image block having the smallest pixel difference between the first sub-image block and the search area corresponding to the first sub-image block in the next image frame.

[0079] Step 2012: The sum of the image block pixel differences corresponding to the first sub-image block at the division scale is used as the image block pixel difference of the preset image block at the division scale.

[0080] In the embodiment of the present application, since the number of first sub-image blocks corresponding to different division scales is also different, here, for each preset image block in each division scale, by counting the image block pixel differences between all first sub-image blocks and the corresponding second sub-image blocks in the preset image block, it is possible to obtain the differences in block matching of different granularities while unifying each division scale into a comparison space of the same size, thereby facilitating the judgment of whether refined matching is needed in subsequent steps.

[0081] Step S202 : determining a target block motion vector of each unit image block in the preset image block based on image block pixel differences at different division scales of the preset image block.

[0082] In some embodiments, for different division scales, the preset image block may be divided into different numbers of first sub-image blocks. Accordingly, for different division scales, the sizes of the corresponding first sub-image blocks are also different.

[0083] For each division scale, the preset image block can be divided into at least one first sub-image block based on the division scale. It is understood that, in the process of calculating the block motion vector corresponding to the first sub-image block, the smaller the size of the first sub-image block obtained based on the division scale, the more the obtained block motion vector focuses on local information of the image; and the larger the size of the first sub-image block obtained based on the division scale, the more the obtained block motion vector focuses on global information of the image.

[0084] The block motion vector corresponding to the unit image block in the first sub-image block under the division scale is the same as the block motion vector corresponding to the first sub-image block.

[0085] Based on this, the embodiment of the present application divides the preset image block into different division scales, and determines the target block motion vector of the unit image block based on the block motion vector of the first sub-image block obtained at each division scale. The target block motion vector of the unit image block obtained in this way not only pays attention to the global information of the image but also pays attention to the local information of the image, thereby improving the accuracy of the target block motion vector.

[0086] In some embodiments, the above step S202 can be implemented through step S2021.

[0087] Step S2021: In response to the image block pixel difference at the first division scale being greater than or equal to the image block pixel difference at any other division scale, for each of the unit image blocks, based on the block motion vector corresponding to the unit image block at each of the division scales, determining a target block motion vector for the unit image block.

[0088] The other division scales include all scales other than the first division scale. When the different division scales include the first division scale and the second division scale, the other division scales include the second division scale. When the different division scales include the first division scale, the second division scale and at least one third division scale, the other division scales include the second division scale and at least one third division scale.

[0089] Here, it is necessary to determine whether the image block pixel difference at the first division scale is smaller than the image block pixel difference at each other division scale. When the image block pixel difference at the first division scale is greater than or equal to the image block pixel difference at any other division scale, for each of the unit image blocks, based on the block motion vector corresponding to the unit image block at each of the division scales, the target block motion vector of the unit image block is determined.

[0090] In an embodiment of the present application, the above can be achieved by: based on the block motion vector corresponding to the unit image block at each division scale, generating the image block pixel difference corresponding to the unit image block at each division scale; and determining the block motion vector corresponding to the minimum image block pixel difference as the target block motion vector of the unit image block.

[0091] For each unit image block in the current image frame, based on different division scales, the block motion vector corresponding to the unit image block is also different.

[0092] In an embodiment of the present application, based on the unit image block and the block motion vector at each division scale, the target image block corresponding to the unit image block can be determined in the next image frame, and the pixel value difference between the unit image block and the target image block is taken pixel by pixel, and the cumulative sum of the pixel value differences of each pixel point is used as the image block pixel difference of the unit image block at each division scale.

[0093] Exemplarily, when the at least two division scales include a first division scale and a second division scale, the pixel difference of the corresponding image block at the first division scale can be determined based on the unit image block and the target image block at the first division scale; and the pixel difference of the corresponding image block at the second division scale can be determined based on the unit image block and the target image block at the second division scale. If the pixel difference of the corresponding image block at the first division scale is smaller than the pixel difference of the corresponding image block at the second division scale, the block motion vector of the unit image block at the first division scale is determined as the target block motion vector of the unit image block; if the pixel difference of the corresponding image block at the first division scale is larger than the pixel difference of the corresponding image block at the second division scale, the block motion vector of the unit image block at the second division scale is determined as the target block motion vector of the unit image block.

[0094] In an embodiment of the present application, by comparing the block motion vectors of the unit image block at different division scales, the block motion vector corresponding to the minimum image block pixel difference is determined as the target block motion vector of the unit image block, which can integrate global information and local information to improve the accuracy of block matching.

[0095] In some embodiments, the above step S202 can be implemented through step S2022.

[0096] Step S2022: In response to the image block pixel difference at the first division scale being smaller than the image block pixel difference at each other division scale, determine the block motion vector corresponding to the preset image block as the target block motion vector of each unit image block in the preset image block.

[0097] In some embodiments, in response to the image block pixel difference at the first division scale being smaller than the image block pixel difference at each other division scale, indicating that the motion estimation at the current first division scale is relatively better, the target block motion vector of each unit image block in the preset image block can be set to the block motion vector corresponding to the preset image block.

[0098] In the embodiment of the present application, by judging whether the pixel difference of the image block at the first division scale is greater than or equal to the pixel difference of the image block at any other division scale, and only when the pixel difference of the image block at the first division scale is greater than or equal to the pixel difference of the image block at any other division scale, the block motion vector corresponding to the unit image block at each of the division scales is compared, and then the target block motion vector of the unit image block is determined. In this way, unnecessary calculation amount can be reduced, and the overall efficiency of obtaining the first motion estimation vector is improved.

[0099] In some embodiments, the method further includes: for each first sub-image block at each of the division scales, determining a search area corresponding to the first sub-image block in the next image frame; the first sub-image block is obtained by dividing the preset image block based on the division scale; within the search area corresponding to the first sub-image block, intercepting the image blocks to be selected with a sliding window corresponding to the size of the first sub-image block, and calculating the image block pixel difference between each of the image blocks to be selected and the first sub-image block; determining the image block corresponding to the smallest image block pixel difference as the second sub-image block corresponding to the first sub-image block; based on the first sub-image block and the corresponding second sub-image block, generating a block motion vector corresponding to the first sub-image block at the division scale; wherein the block motion vector corresponding to the unit image block within the first sub-image block at the division scale is the same as the block motion vector corresponding to the first sub-image block.

[0100] In some embodiments, the search area corresponding to the first sub-image block is an area centered on the position coordinates of the first sub-image block. For example, taking the search area as a rectangular area, where the position coordinates of the first sub-image block are (x, y) in the current image frame, the search area corresponding to the first sub-image block is a rectangular area from (xm, yn) to (x+m, y+n) in the next image frame.

[0101] In an embodiment of the present application, the corresponding image blocks to be selected are sequentially intercepted from the next image frame using the sliding window within the search area corresponding to the first sub-image block, and the difference of pixel values ​​is taken pixel by pixel. The sum of the differences of the pixel values ​​of each pixel in the sliding window is taken as the image block pixel difference between the image block to be selected and the first sub-image block.

[0102] It can be understood that in the embodiment of the present application, the sliding step size of the sliding window can be adaptively set based on the calculation accuracy, with the minimum setting being one pixel.

[0103] In this embodiment of the present application, after obtaining the image block pixel difference between each candidate image block in the search area and the first sub-image block, the candidate image block corresponding to the minimum image block pixel difference is determined as the second sub-image block corresponding to the first sub-image block.

[0104] Here, the vector pointing from the position coordinates of the first sub-image block to the position coordinates of the second sub-image block is used as the block motion vector corresponding to the first sub-image block under the division scale.

[0105] In the above embodiment, the position coordinates of the image block can be the relative coordinates of the center point of the image block in the image frame, or the relative coordinates of the upper left corner point of the image block in the image frame. Of course, it can also be other relative coordinates that represent the relative position relationship between the image block and the image frame. This application does not limit this.

[0106] In the embodiment of the present application, by setting a search area and performing block matching on the first sub-image block in the form of a sliding window within the search area, the matching accuracy can be improved while improving the matching efficiency.

[0107] FIG3A is a third optional flow diagram of a dense optical flow generation method provided in an embodiment of the present application, which can be executed by a processor of a computer device. Based on FIG1 , S102 in FIG1 can be updated to S301 to S302 , which will be described in conjunction with the steps shown in FIG3A .

[0108] Step S301: For each pixel to be processed, obtain a plurality of neighboring pixel points corresponding to the pixel to be processed in the current image frame, and obtain a plurality of neighboring pixel points corresponding to the matching pixel in the next image frame.

[0109] In an embodiment of the present application, the multiple neighborhood pixel points corresponding to the pixel point to be processed are evenly distributed around the pixel point to be processed, and correspondingly, the multiple neighborhood pixel points corresponding to the matching pixel point are evenly distributed around the matching pixel point.

[0110] It can be understood that the number of the plurality of neighboring pixel points corresponding to the pixel point to be processed is the same as the number of the plurality of neighboring pixel points corresponding to the matching pixel point, and there is a one-to-one correspondence between them.

[0111] For example, refer to FIG. 3B , which illustrates a schematic diagram of the relationship between neighboring pixels in a current image frame and a next image frame. Here, a pixel to be processed P10 in a current image frame 31 is matched with a matching pixel P20 in a next image frame 32 , and the matching pixel P20 can be determined based on the pixel to be processed P10 and the corresponding first motion estimation vector 33 . The multiple neighborhood pixel points (6 in the figure, namely P11 to P16) corresponding to the pixel point P10 to be processed are evenly distributed around the pixel point P10 to be processed. At the same time, the multiple neighborhood pixel points (6 in the figure, namely P21 to P26) corresponding to the matching pixel point P20 are evenly distributed around the matching pixel point P20. At the same time, the neighborhood pixel point P11 in the current image frame 31 corresponds to the neighborhood pixel point P21 in the next image frame 32, and the neighborhood pixel point P12 in the current image frame 31 corresponds to the neighborhood pixel point P22 in the next image frame 32, and so on. That is, there is a one-to-one correspondence between the multiple neighborhood pixel points corresponding to the pixel point to be processed and the multiple neighborhood pixel points corresponding to the matching pixel point.

[0112] It is understandable that the selection method of the above-mentioned multiple neighborhood pixel points can be adaptively configured based on the specific implementation scenario, and the embodiment of the present application does not limit the position of the neighborhood pixel points.

[0113] In this embodiment, by respectively obtaining multiple neighborhood pixel points around the pixel to be processed and multiple neighborhood pixel points around the matching pixel point, not only can the image information of the local image of the current pixel point be reflected through several neighborhood pixel points, but also, compared with the scheme in the related art that requires multiple iterations in the process of matching the local image where the current pixel point is located, the embodiment of the present application can reduce the amount of calculation in the subsequent second motion estimation vector determination process.

[0114] In some embodiments, the number of neighborhood pixels may be determined based on the type of optical flow method used in the implementation. For example, when fitting the second motion estimation vector corresponding to the pixel to be processed based on the Farneback optical flow method, at least 6 neighborhood pixels are required.

[0115] Step S302: Fitting a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a plurality of neighboring pixels corresponding to the pixel to be processed and a plurality of neighboring pixels corresponding to the matching pixel.

[0116] In some embodiments, the association between texture information and position coordinates in the local image where the pixel to be processed is located can be extracted based on multiple neighborhood pixel points corresponding to the pixel to be processed; at the same time, the association between texture information and position coordinates in the local image where the matching pixel is located can be extracted based on multiple neighborhood pixel points corresponding to the matching pixel; based on the matching relationship between the texture information in the local image where the pixel to be processed is located and the texture information in the local image where the matching pixel is located, a second motion estimation vector of the pixel to be processed is generated.

[0117] Exemplarily, when fitting the second motion estimation vector corresponding to the pixel to be processed based on the Farneback optical flow method, a corresponding quadratic polynomial function can be constructed based on multiple neighborhood pixel points corresponding to the pixel to be processed, and a corresponding quadratic polynomial function can be constructed based on multiple neighborhood pixel points corresponding to the matching pixel point; since these two quadratic polynomial functions both represent the association relationship between image information and position coordinates, and the image information of the local image where the pixel to be processed is located is associated with the image information in the local image where the matching pixel point is located by the second motion estimation vector, the second motion estimation vector of the pixel to be processed can be fitted based on the two obtained quadratic polynomial functions.

[0118] In an embodiment of the present application, by respectively obtaining multiple neighborhood pixel points around the pixel to be processed and multiple neighborhood pixel points around the matching pixel point, the image information of the local image of the current pixel point can be reflected through several neighborhood pixel points; at the same time, compared with the scheme in the related art that requires multiple iterations in the process of matching the local image where the current pixel point is located, the embodiment of the present application fits the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between the multiple neighborhood pixel points corresponding to the pixel to be processed and the multiple neighborhood pixel points corresponding to the matching pixel point, thereby reducing the process of multiple iterative calculations, reducing the calculation amount of optical flow estimation, and shortening the calculation time.

[0119] In some embodiments, steps S3021 to S3023 can be used to implement the above-mentioned matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame, and fit the second motion estimation vector corresponding to each pixel to be processed.

[0120] Step S3021: construct a first matrix function based on a plurality of neighboring pixel points corresponding to the pixel point to be processed.

[0121] Among them, the first matrix function includes the first pixel function of each neighboring pixel point corresponding to the pixel point to be processed, and the first pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighboring pixel points in the current image frame and the grayscale values ​​of the neighboring pixel points.

[0122] In an embodiment of the present application, it is necessary to obtain the grayscale value and position coordinates corresponding to each of the neighboring pixel points corresponding to the pixel point to be processed, and construct a quadratic polynomial function corresponding to each of the neighboring pixel points corresponding to the pixel point to be processed as the first pixel function of the neighboring pixel points corresponding to the pixel point to be processed, as shown in formula (1).

[0123] f(X)~X T AX+b T X+c formula (1);

[0124] Where f(X) is the pixel value of the neighboring pixel corresponding to the pixel to be processed, A is a 2×2 matrix, b is a 2×1 matrix, X is the coordinates (x, y) of the neighboring pixel corresponding to the pixel to be processed, and c is a scalar. Based on this, Formula (1) can be written as Formula (2).

[0125] Where A is a 2×2 matrix b is a 2×1 matrix c is r1.

[0126] It can be seen that in order to determine the correlation between the texture information and the position coordinates in the local image where the pixel to be processed is located, at least 6 neighborhood pixel points corresponding to the pixel to be processed are required, and then 6 quadratic polynomial functions, namely the first matrix function, are constructed. By solving the first matrix function, the first pixel function corresponding to the pixel to be processed is obtained.

[0127] Step S3022: construct a second matrix function based on multiple neighborhood pixel points corresponding to the neighborhood pixel point.

[0128] Among them, the second matrix function includes a second pixel function of each neighborhood pixel point corresponding to the neighborhood pixel point, and the second pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighborhood pixel point in the next image frame and the grayscale value of the neighborhood pixel point.

[0129] In an embodiment of the present application, it is necessary to obtain the grayscale value and position coordinates corresponding to each of the neighborhood pixel points corresponding to the matching pixel point, and construct a quadratic polynomial function corresponding to each of the neighborhood pixel points corresponding to the matching pixel point as the second pixel function of the neighborhood pixel point corresponding to the matching pixel point, as shown in formula (3).

[0130] f′(X′)~X′ T A′X′+b′ T X′+c′ formula (3);

[0131] Where f′(X′) is the pixel value of the neighboring pixel corresponding to the matching pixel, A′ is a 2×2 matrix, b′ is a 2×1 matrix, X′ is the coordinates (x′, y′) of the neighboring pixel corresponding to the matching pixel, and c′ is a scalar.

[0132] Similar to formula (2), in order to determine the correlation between the texture information and the position coordinates in the local image where the matching pixel point is located, at least 6 neighboring pixel points corresponding to the matching pixel point are required (the same number as the neighboring pixel points corresponding to the pixel point to be processed), and then 6 quadratic polynomial functions, namely the second matrix function, are constructed. By solving the second matrix function, the second pixel function corresponding to the matching pixel point is obtained.

[0133] Step S3023: Generate a second motion estimation vector corresponding to the pixel to be processed based on a matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function.

[0134] In an embodiment of the present application, the first pixel function represents the association relationship between the current local image information and the position coordinates in the current image frame, and the second pixel function represents the association relationship between the current local image information and the position coordinates in the next image frame; based on the fact that the current local image information in the current image frame is the same as the current local image information in the next image frame, and there is a coordinate offset, a relationship between the first pixel function and the second pixel function can be constructed to obtain the coordinate offset, that is, the second motion estimation vector corresponding to the pixel point to be processed.

[0135] X = X′-d formula (4);

[0136] Where X is the coordinate of the pixel in the current image frame, X′ is the coordinate of the pixel in the next image frame, and d is the second motion estimation vector. The relationship between the first pixel function and the second pixel function can be expressed as formula (5):

[0137] f(X)=f′(X′-d) Formula (5);

[0138] In this embodiment, the Farneback optical flow method is used to construct a first pixel function that represents the relationship between the current local image information and the position coordinates in the current image frame, and a second pixel function that represents the relationship between the current local image information and the position coordinates in the next image frame. The coordinate offset is then solved to obtain a second motion estimation vector corresponding to the pixel to be processed. This reduces the computational complexity and computation time of the optical flow estimation compared to the traditional Farneback optical flow method, which requires multiple iterations for all pixels in the local image.

[0139] In some embodiments, the quadratic polynomial coefficients of the pixel function include matrix coefficients and vector coefficients; the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function is used to generate the second motion estimation vector corresponding to the pixel point to be processed, including: determining the mean matrix coefficient based on the first matrix coefficient corresponding to the first pixel function and the second matrix coefficient corresponding to the second pixel function; determining the difference vector coefficient based on the first vector coefficient corresponding to the first pixel function and the second vector coefficient corresponding to the second pixel function; and determining the product of the inverse matrix corresponding to the mean matrix coefficient, the preset coefficient and the difference vector coefficient as the second motion estimation vector corresponding to the pixel point to be processed.

[0140] Wherein, the matrix coefficient is A in the above formula (1), and the vector coefficient is b in the above formula (1). To facilitate understanding of the above method for determining the second motion estimation vector, please refer to the following derivation process: In the embodiment of the present application, by transforming formula (5), formula (6) can be obtained.

[0141] f′(X′-d)=(X′-d) T A′(X′-d)+b′ T (X′-d)+c′=X T A′X+(b′-2A′d) T X+d T Ad-b′ T d+c′ formula (6);

[0142] Since the parameter A in the quadratic polynomial function represents the image information, the A corresponding to the pixel to be processed in the current image frame should be the same as the A′ corresponding to the matching pixel in the next image frame. The coordinate offset d can be expressed as formula (7);

[0143] Among them, A″ can be the first matrix coefficient corresponding to the first pixel function, or the second matrix coefficient corresponding to the second pixel function. In order to improve the accuracy of image information, the average of the first matrix coefficient corresponding to the first pixel function and the second matrix coefficient corresponding to the second pixel function can be taken as the mean matrix coefficient. (b′-b) ​​is the difference between the first vector coefficient corresponding to the first pixel function and the second vector coefficient corresponding to the second pixel function, that is, the difference vector coefficient, and 1 / 2 is the preset coefficient; furthermore, the coordinate offset d can be expressed as the product of the inverse matrix corresponding to the mean matrix coefficient, the preset coefficient and the difference vector coefficient.

[0144] Among them, considering that A may not be a singular matrix and cannot be directly inverted, we can modify formula (7) to obtain formula (8), and then obtain the motion estimation vector of the current pixel (that is, the local optical flow offset).

[0145] d=(A″ T A) -1 A″T*(b′-b) / 2 Formula (8).

[0146] In the embodiment of the present application, the second motion estimation vector corresponding to the pixel to be processed can be quickly obtained through matrix operation, thereby improving calculation efficiency.

[0147] FIG4 is a fourth optional flow diagram of a dense optical flow generation method provided in an embodiment of the present application, which can be executed by a processor of a computer device. Based on FIG1 , the dense optical flow information includes a target motion estimation vector corresponding to each pixel to be processed; S103 in FIG1 can be updated to S401 to S402, which will be explained in conjunction with the steps shown in FIG4 .

[0148] Step S401: For each pixel to be processed, based on a preset pixel difference calculation window, determine the first calculation window pixel difference corresponding to the pixel to be processed under the first motion estimation vector, and the second calculation window pixel difference corresponding to the pixel to be processed under the second motion estimation vector.

[0149] In some embodiments, the pixel difference calculation window is a window centered on the pixel to be processed. The shape of the pixel difference calculation window can be rectangular, square, or circular, which is not limited in the embodiments of the present application. For example, taking the shape of the pixel difference calculation window as a square, the size of the pixel difference calculation window can be (2n+1)*(2n+1), and accordingly, the center point of the pixel to be processed in the corresponding pixel difference calculation window, that is, the relative coordinates in the pixel difference calculation window are (n+1, n+1).

[0150] In an embodiment of the present application, based on the first motion estimation vector and the pixel point to be processed, the first target pixel point corresponding to the pixel point to be processed can be determined in the next image frame; based on the pixel point to be processed, a corresponding pixel difference calculation window is constructed in the current image frame, and based on the first target pixel point, a corresponding pixel difference calculation window is constructed in the next image frame; the pixel difference calculation window corresponding to the pixel point to be processed and the pixel difference calculation window corresponding to the first target pixel point are subjected to pixel-by-pixel difference calculations, and the sum of the differences in the pixel values ​​of each pixel point in the pixel difference calculation window is taken as the pixel difference of the first calculation window corresponding to the pixel point to be processed.

[0151] Exemplarily, taking the pixel difference calculation window as a square window with a size of (2n+1)*(2n+1) as an example, based on the pixel point to be processed and the pixel difference calculation window, (2n+1)*(2n+1) pixel points are intercepted in the current image frame. Correspondingly, based on the first target pixel point and the pixel difference calculation window, (2n+1)*(2n+1) pixel points are intercepted in the next image frame. After taking the difference of pixel values ​​pixel by pixel, the difference of the pixel values ​​of (2n+1)*(2n+1) pixel points can be obtained, and after accumulation, the pixel difference of the first calculation window corresponding to the pixel point to be processed can be obtained.

[0152] In some embodiments, the second calculation window pixel difference is determined based on the same calculation method as the first calculation window pixel difference. Based on the second motion estimation vector and the pixel to be processed, a second target pixel corresponding to the pixel to be processed can be determined in the next image frame; a corresponding pixel difference calculation window is constructed in the current image frame based on the pixel to be processed, and a corresponding pixel difference calculation window is constructed in the next image frame based on the second target pixel; the pixel difference calculation window corresponding to the pixel to be processed and the pixel difference calculation window corresponding to the second target pixel are subjected to pixel-by-pixel difference calculation, and the sum of the differences in the pixel values ​​of the pixels within the pixel difference calculation window is used as the pixel difference of the second calculation window corresponding to the pixel to be processed.

[0153] Step S402: Using the motion estimation vector corresponding to the minimum calculation window pixel difference between the first calculation window pixel difference and the second calculation window pixel difference as the target motion estimation vector corresponding to the pixel point to be processed.

[0154] In some embodiments, when the first calculation window pixel difference is smaller than the second calculation window pixel difference, that is, the first calculation window pixel difference is the minimum calculation window pixel difference, the first motion estimation vector is used as the target motion estimation vector corresponding to the pixel to be processed.

[0155] In some embodiments, when the first calculation window pixel difference is greater than the second calculation window pixel difference, that is, the second calculation window pixel difference is the minimum calculation window pixel difference, the second motion estimation vector is used as the target motion estimation vector corresponding to the pixel to be processed.

[0156] In some embodiments, when the first calculation window pixel difference is equal to the second calculation window pixel difference, any one of the first motion estimation vector and the second motion estimation vector can be used as the target motion estimation vector corresponding to the pixel to be processed.

[0157] In the embodiment of the present application, for each pixel to be processed, the first calculation window pixel difference corresponding to the pixel to be processed under the first motion estimation vector and the second calculation window pixel difference corresponding to the pixel to be processed under the second motion estimation vector are calculated, and the final target motion estimation vector for the pixel to be processed is determined by comparing the two results. In this way, the target motion estimation vector obtained for each pixel to be processed not only takes into account global image information but also local image information. At the same time, the comparison results of the calculation window pixel differences of the two motion estimation vectors are combined to improve the accuracy of the target motion estimation vector for each pixel to be processed, thereby improving the accuracy of the dense optical flow.

[0158] The following describes the application of the dense optical flow generation method provided in the embodiments of the present application in actual scenarios.

[0159] In the field of video image processing, algorithms such as object tracking, image registration, High Dynamic Range Imaging (HDR) synthesis, and multi-frame noise reduction all require dense optical flow. Numerous methods exist for calculating dense optical flow, including block matching, local binarization approximation, energy minimization-based optical flow algorithms, and even deep learning network training methods. However, these methods all require significant computational effort.

[0160] In some embodiments, the calculation of the dense optical flow of the above-mentioned image can be completed by a block matching method. The block matching method divides the image into multiple image blocks, and in adjacent frames, searches within a fixed or variable neighborhood based on this image block, compares the image block in the current frame with the image blocks in the neighborhood in the adjacent frame, and performs a difference. The image block with the smallest difference in the neighborhood in the adjacent frame is selected, and the coordinate difference of its current image block is recorded, which is the optical flow vector of the current image block. The advantage of this method is low computational complexity, but the disadvantage is that it requires a large amount of computation and has low matching accuracy.

[0161] In other embodiments, the calculation of the dense optical flow of the above image can also be completed by a local binarization approximation algorithm. This method assumes that the local image block can be expressed by a quadratic polynomial function, such as formula (9).

[0162] f(x)~x T Ax+b T x+c formula (9);

[0163] Where A is a 2×2 matrix, b is a 2×1 matrix, x is the coordinates of the image block (xy), and c is a scalar. Based on this, formula (9) can be written as formula (10).

[0164] Among them, there are 6 unknowns in formula (10) (including r1 to r6). Therefore, to obtain this equation and then represent the local image block, at least 6 local image blocks of information are required. According to the assumption of optical flow, the same image blocks have the same image information, and adjacent image blocks have similar image information. Therefore, the current local image block and the 6 local image blocks in the surrounding neighborhood can be used as a group to fit the above equation. Therefore, formula (10) can be converted into (1, x, y, x 2 ,u 2 ,xy) is the matrix function of the basis, as shown in formula (11).

[0165] For the current local image block, the position coordinate difference between two adjacent frames of images can be expressed as formula (12).

[0166] x=x+d(x) Formula (12);

[0167] Where x is the position coordinate and d(x) is the displacement; then, the intermediate variables Δb(x) and h(x) can be obtained and determined by formula (13) and formula (14), respectively.

[0168] Δb(x)=b2(x)-b1(x)+A(x)d(x) Formula (13);

[0169] h(x)=A(x) T *Δb(x) Formula (14);

[0170] Finally, the optical flow field d can be solved based on the intermediate variables Δb(x) and h(x) out (x), as shown in formula (15).

[0171] d out (x) = G avg (x) -1 *h avg(x) Formula (15);

[0172] The above algorithm iterates until the number of iterations is reached. The advantage of binarization is that it does not rely on image texture information and can calculate motion estimation vectors for very small image blocks or even pixels. The disadvantage is that the number of binarization iterations is large and, due to the lack of image texture support, problems may occur in noisy areas and areas with fine textures.

[0173] Both block matching and methods based on solving local image equations have the problem of excessive computational complexity. For block matching, there is an inability to estimate finer pixel-level motion vectors; for solving local image equations, there is a lack of image texture information, sensitivity to noise, and the need for recursive iteration of a single computing unit, making hardware pipeline implementation difficult. Due to the above problems, the dense optical flow generation method provided in the above embodiment is difficult to achieve both real-time and effective results in products.

[0174] This embodiment of the present application reduces the computational effort required to calculate the initial motion estimation vector for dense optical flow calculations by reusing the intermediate results of existing encoder modules. Furthermore, for small local blocks or pixels, a refined motion estimation vector is calculated using image binarization approximation. This method reduces the computational effort required for optical flow estimation and shortens computation time by reusing modules. It also shortens the computational path for image binarization approximation, improving hardware timeliness.

[0175] This embodiment of the application passes the codec's motion estimation vector information to a dense optical flow calculation module (corresponding to the dense optical flow generation portion in the above embodiment). The dense optical flow calculation module uses the encoder's motion estimation vector information as a basis for image binarization approximation calculations. The calculated results are then compared with the motion estimation vector information provided by the codec using block-by-block difference, and the motion estimation vector with the smallest image block difference is selected as the final result. This process solves the problems of large image dense optical flow calculation complexity, poor real-time performance, and the inability to implement it in hardware.

[0176] Please refer to Figure 5, which shows a data flow diagram of a dense optical flow calculation system. The present embodiment provides a video codec 510 and a dense optical flow calculation module 520. The video codec 510 is used to generate motion estimation vectors corresponding to two adjacent image frames based on unit image blocks; the dense optical flow calculation module 520 is used to perform localized binarization optical flow calculations based on the two adjacent image frames and the corresponding motion estimation vectors to obtain pixel-by-pixel motion estimation vectors. The unit image block size can be set to 8×8.

[0177] In some embodiments, the minimum processing unit of the video codec 510 is a 32×32 image block (corresponding to the preset image block in the above embodiment). The video codec 510 can calculate motion estimation vectors at different scales, and then determine the motion estimation vector corresponding to each unit image block in the 32×32 image block.

[0178] In the embodiments of the present application, in order to derive motion estimation vectors as accurately as possible, it is necessary to compare the motion estimation vectors of unit image blocks at the same location and at different scales. Because the motion estimation vector used as optical flow needs to take into account the continuity of adjacent image regions, before comparing different scales, it is necessary to compare whether the 32×32 blocks are divided.

[0179] Please refer to Figure 6, which shows a schematic diagram of motion estimation of a unit image block. The motion estimation vector of the image block in this embodiment corresponds to the block motion vector in the above embodiment.

[0180] Step S601: Calculate the image block pixel differences of 32×32 image blocks at different scales.

[0181] As shown in FIG6 , the current embodiment includes three division scales (which may be referred to as scales for short): 32×32, 16×16, and 8×8. Therefore, it is necessary to calculate the image block pixel difference at the 32×32 scale, the image block pixel difference at the 16×16 scale, and the image block pixel difference at the 8×8 scale, respectively.

[0182] Taking the 16×16 scale as an example, the pixel difference of a 32×32 image block at this scale is determined by accumulating the pixel differences of four 16×16 image blocks. For any 16×16 image block, the 16×16 image block in the current frame is obtained, and multiple 16×16 image blocks within the window containing the image block in the adjacent frame are obtained. The sum of absolute differences (SAD) between the 16×16 image block in the current frame and each 16×16 image block within the window containing the image block in the adjacent frame is calculated. The minimum sum of absolute differences is used as the pixel difference of the 16×16 image block.

[0183] Based on the same method, the pixel difference of the image block at the scale of 32×32, the pixel difference of the image block at the scale of 16×16, and the pixel difference of the image block at the scale of 8×8 can be obtained.

[0184] Step S602 : Whether the pixel difference of the image block at the 32×32 scale is smaller than the pixel difference of the image block at the 8×8 scale and the pixel difference of the image block at the 16×16 scale.

[0185] In the embodiment of the present application, the size relationship between the image block pixel difference at the 32×32 scale, the image block pixel difference at the 16×16 scale, and the image block pixel difference at the 8×8 scale is compared. If the image block pixel difference at the 32×32 scale is smaller than the image block pixel difference at the 16×16 scale, and the image block pixel difference at the 32×32 scale is smaller than the image block pixel difference at the 8×8 scale, it is determined that no detailed division is required and step S603 is executed; otherwise, it is determined that detailed division is required and step S604 is executed.

[0186] Step S603 : The motion estimation vector of each 8×8 unit image block in the 32×32 image block is derived from the motion estimation vector of the 32×32 image block.

[0187] Step S604 : The motion estimation vector of each 8×8 unit image block in the 32×32 image block is determined based on the motion estimation vectors of different scales corresponding to each 8×8 unit image block.

[0188] In an embodiment of the present application, when a 32×32 image block needs to be further divided, the 32×32 image block needs to be divided into 8×8 unit image blocks. Each unit image block corresponds to three types of motion estimation vectors: a motion estimation vector at an 8×8 scale, a motion estimation vector at a 16×16 scale, and a motion estimation vector at a 32×32 scale. The pixel differences of the 8×8 image block under these three motion estimation vectors are compared simultaneously, and the motion estimation vector at the scale with the smallest difference is selected as the output motion estimation vector for the 8×8 image block.

[0189] Please refer to FIG. 7 , which shows a flow chart of motion estimation of a unit image block in a refined division state.

[0190] Step S701 : For each unit image block, determine the motion estimation vector of the unit image block at different scales.

[0191] Step S702: Determine the image block pixel difference corresponding to the motion estimation vector of the unit image block at each scale.

[0192] The motion estimation vector of the unit image block at an 8×8 scale and the corresponding image block pixel difference are obtained in the following manner: obtaining the sum of absolute differences (SAD) between the unit image block in the current frame and each 8×8 image block within the window where the unit image block is located in the adjacent frame, determining the minimum absolute error and corresponding image block in the adjacent frame, and then generating the motion estimation vector of the unit image block at an 8×8 scale based on the position between the minimum absolute error and the corresponding image block in the adjacent frame and the unit image block in the current frame, and using the minimum absolute error as the image block pixel difference of the unit image block at an 8×8 scale.

[0193] The motion estimation vector of the unit image block at a 16×16 scale and the corresponding image block pixel difference are obtained in the following manner: determining the 16×16 image block in the current frame where the unit image block is located, obtaining multiple 16×16 image blocks within a window where the 16×16 image block is located in adjacent frames, calculating the sum of absolute errors between the 16×16 image block in the current frame and each 16×16 image block within the window where the image block is located in adjacent frames, determining the 16×16 image block corresponding to the minimum absolute error sum in adjacent frames, and then generating a motion estimation vector for the unit image block at a 16×16 scale based on the position between the 16×16 image block corresponding to the minimum absolute error sum in adjacent frames and the 16×16 image block in the current frame, i.e., the motion estimation vector at a 16×16 scale; determining the initial image block corresponding to the unit image block in adjacent frames using the motion estimation vector; and determining the sum of absolute errors between the unit image block and the initial image block as the image block pixel difference of the unit image block at a 16×16 scale.

[0194] Among them, based on the same calculation method as that of the 16×16 scale, the motion estimation vector and image block pixel difference of the unit image block at the 32×32 scale can be obtained.

[0195] Step S703: Use the motion estimation vector corresponding to the minimum image block pixel difference as the motion estimation vector of the unit pixel block.

[0196] In this embodiment, the image block pixel differences of the unit image block at 8×8 scale, 16×16 scale, and 32×32 scale are compared, and the motion estimation vector corresponding to the minimum image block pixel difference is used as the motion estimation vector of the unit pixel block.

[0197] Based on the above embodiment, the motion estimation vector corresponding to each unit image block can be obtained. It can be understood that the motion estimation vector of each pixel in the unit image block is the same as the motion estimation vector corresponding to the unit image block, and thus the motion estimation vector of each pixel can be obtained.

[0198] Please refer to Figure 8, which shows a schematic diagram of a dense optical flow motion estimation vector. The dense optical flow calculation module uses the motion estimation vector of the 8×8 image block output by the video codec 510 as the basis, and performs a pixel-by-pixel image binarization approximation on this basis. As shown in Figure 8, the local image binarization approximation is based on the coordinates (x1, y1) of image block 2, and each pixel within image block 2 is processed as follows:

[0199] The pixel to be solved in the reference image 820 is selected as the center, and six points in the neighborhood are selected to form a matrix function for binomial calculation, as shown in formula (16). For example, the six points in the neighborhood can be selected according to the neighborhood point schematic diagram shown in Figure 9.

[0200] Where f(x) is the grayscale value of the pixel to be solved, (x, y) is the coordinate of the pixel to be solved, and r1-r6 are the coefficients of the binomial. Another form of the binomial is formula (17).

[0201] Where A is a 2×2 matrix b is a 2×1 matrix c is r1.

[0202] For pixels with the same image information, A is the same. Therefore, the A of a pixel in the current image 810 is the same as the A of a pixel with the same image information in the reference image 820. The pixel function in the reference image 820 can be expressed as formula (18):

[0203] Therefore, the offset of pixel point A between the current image 810 and the reference image 820, the offset d, can satisfy formula (19).

[0204] Since A may not be a singular matrix and cannot be directly inverted, formula (20) can be modified to obtain formula (12), and then the motion estimation vector of the current pixel corresponding to the pixel to be solved (i.e., the local optical flow offset) is obtained.

[0205] d=(A T A) -1 A T *(b-b1) / 2 Formula (20);

[0206] In some embodiments, for calculating the motion estimation vector of the current pixel, the embodiment of the present application determines the coordinates of the reference pixel corresponding to the current pixel in the reference image 820 based on the motion estimation vector corresponding to the unit image block where the current pixel is located, uses a 7×7 area centered on the current pixel in the current image as a calculation window, and uses a 7×7 area centered on the reference pixel in the reference image as another corresponding calculation window, performs pixel-by-pixel difference on the two calculation windows, and statistically calculates the sum of the pixel differences as the first calculation window pixel difference; the embodiment of the present application also determines the coordinates of the reference pixel corresponding to the current pixel in the reference image based on the motion estimation vector of the current pixel determined by formula (20), uses a 7×7 area centered on the current pixel in the current image as a calculation window, and uses a 7×7 area centered on the reference pixel in the reference image as another corresponding calculation window, performs pixel-by-pixel difference on the two calculation windows, and statistically calculates the sum of the pixel differences as the second calculation window pixel difference. The pixel difference of the first calculation window is compared with the pixel difference of the second calculation window, and the motion estimation vector with the smaller pixel difference of the calculation window is used as the target motion estimation vector of the current pixel.

[0207] This embodiment of the application transmits the video codec's motion estimation vector to a dense optical flow calculation module. Based on the video codec's motion estimation vector, the image binomial equation is fitted in the neighborhood of the pixel to be determined, combined with six-pixel information, and the local pixel offset is calculated using the least squares method. The motion estimation vector corresponding to the local pixel offset is then compared with the video codec's motion estimation vector for the local image block pixel difference, and the motion estimation vector with the smallest difference is selected as the target motion estimation vector for the current pixel.

[0208] Based on the above embodiment, by using the motion estimation vector of the video codec as a basis, the complexity of pyramid calculation of image optical flow using dense optical flow fields is reduced, and the optical flow calculation time is reduced. At the same time, based on the motion estimation vector of the video codec, local optical flow calculation based on image binarization function fitting is performed, and local dense optical flow is obtained through matrix operations without multiple iterations, which reduces the data flow path of hardware processing and improves hardware feasibility. At the same time, by comparing the optical flow using the motion estimation vector of the video codec and the optical flow using the image binarization function fitting, the instability of the image binarization function fitting method with respect to noise and fine textures is reduced, thereby improving the accuracy of the dense optical flow.

[0209] Based on the foregoing embodiments, an embodiment of the present application provides a dense optical flow generation device, which includes the various units included and the various modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0210] FIG10 is a schematic diagram of the structure of a dense optical flow generating device provided in an embodiment of the present application. As shown in FIG10 , the dense optical flow generating device 1000 includes: a first vector acquisition part 1010, a second vector acquisition part 1020, and a dense optical flow generating part 1030, wherein:

[0211] The first vector acquisition section 1010 is configured to perform block matching on the current image frame and the next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; the first motion estimation vector corresponding to the pixel to be processed is determined by the block matching result corresponding to the unit image block where the pixel to be processed is located;

[0212] The second vector acquisition part 1020 is configured to fit a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a partial image of each pixel to be processed in the current image frame and a partial image of a corresponding matching pixel in the next image frame; the matching pixel is determined based on the pixel to be processed and the corresponding first motion estimation vector;

[0213] The dense optical flow generating part 1030 is configured to generate dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed.

[0214] In some embodiments, the first motion estimation vector corresponding to the pixel to be processed is the target block motion vector of the unit image block where the pixel to be processed is located; the first vector acquisition part 1010 is further configured to: determine the image block pixel difference of the preset image block at different division scales for each preset image block in the current image frame; and determine the target block motion vector of each unit image block in the preset image block based on the image block pixel difference of the preset image block at different division scales.

[0215] In some embodiments, the different division scales include at least a first division scale corresponding to the preset image block, and the first vector acquisition part 1010 is further configured to: in response to the image block pixel difference at the first division scale being greater than or equal to the image block pixel difference at any other division scale, determine, for each of the unit image blocks, the target block motion vector of the unit image block based on the block motion vector corresponding to the unit image block at each of the division scales.

[0216] In some embodiments, the different division scales include at least a first division scale corresponding to the preset image block, and the first vector acquisition part 1010 is further configured to: in response to the image block pixel difference at the first division scale being smaller than the image block pixel difference at each other division scale, determine the block motion vector corresponding to the preset image block as the target block motion vector of each unit image block in the preset image block.

[0217] In some embodiments, the first vector acquisition part 1010 is further configured to: determine, for each of the division scales, the image block pixel difference between the first sub-image block and the corresponding second sub-image block at the division scale; the first sub-image block is obtained by dividing the preset image block based on the division scale; the second sub-image block is the image block in the next image frame with the smallest image block pixel difference with the first sub-image block; and the sum of the image block pixel differences corresponding to the first sub-image block at the division scale is used as the image block pixel difference of the preset image block at the division scale.

[0218] In some embodiments, the first vector acquisition part 1010 is further configured to: generate an image block pixel difference corresponding to the unit image block at each division scale based on the block motion vector corresponding to the unit image block at each division scale; and determine the block motion vector corresponding to the minimum image block pixel difference as the target block motion vector of the unit image block.

[0219] In some embodiments, the first vector acquisition part 1010 is further configured to: for each first sub-image block at each of the division scales, determine a search area corresponding to the first sub-image block in the next image frame; the first sub-image block is obtained by dividing the preset image block based on the division scale; within the search area corresponding to the first sub-image block, intercept the image blocks to be selected with a sliding window corresponding to the size of the first sub-image block, and calculate the image block pixel difference between each of the image blocks to be selected and the first sub-image block; determine the image block to be selected corresponding to the smallest image block pixel difference as the second sub-image block corresponding to the first sub-image block; based on the first sub-image block and the corresponding second sub-image block, generate a block motion vector corresponding to the first sub-image block at the division scale; wherein the block motion vector corresponding to the unit image block within the first sub-image block at the division scale is the same as the block motion vector corresponding to the first sub-image block.

[0220] In some embodiments, the second vector acquisition part 1020 is further configured to: for each pixel point to be processed, obtain multiple neighborhood pixel points corresponding to the pixel point to be processed in the current image frame, and obtain multiple neighborhood pixel points corresponding to the matching pixel point in the next image frame; based on the matching relationship between the multiple neighborhood pixel points corresponding to the pixel point to be processed and the multiple neighborhood pixel points corresponding to the matching pixel point, fit the second motion estimation vector corresponding to each pixel point to be processed.

[0221] In some embodiments, the second vector acquisition part 1020 is further configured to: construct a first matrix function based on multiple neighborhood pixel points corresponding to the pixel point to be processed; the first matrix function includes a first pixel function of each neighborhood pixel point corresponding to the pixel point to be processed, and the first pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighborhood pixel point in the current image frame and the grayscale value of the neighborhood pixel point; construct a second matrix function based on multiple neighborhood pixel points corresponding to the neighborhood pixel point; the second matrix function includes a second pixel function of each neighborhood pixel point corresponding to the neighborhood pixel point, and the second pixel function is a quadratic polynomial function that characterizes the correlation relationship between the position coordinates of the neighborhood pixel point in the next image frame and the grayscale value of the neighborhood pixel point; based on the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function, generate a second motion estimation vector corresponding to the pixel point to be processed.

[0222] In some embodiments, the quadratic polynomial coefficients of the pixel function include matrix coefficients and vector coefficients; the second vector acquisition part 1020 is also configured to: determine the mean matrix coefficients based on the first matrix coefficients corresponding to the first pixel function and the second matrix coefficients corresponding to the second pixel function; determine the difference vector coefficients based on the first vector coefficients corresponding to the first pixel function and the second vector coefficients corresponding to the second pixel function; and determine the product of the inverse matrix corresponding to the mean matrix coefficients, the preset coefficients and the difference vector coefficients as the second motion estimation vector corresponding to the pixel to be processed.

[0223] In some embodiments, the dense optical flow information includes a target motion estimation vector corresponding to each of the pixel points to be processed; the dense optical flow generation part 1030 is further configured to: for each of the pixel points to be processed, based on a preset pixel difference calculation window, determine the first calculation window pixel difference corresponding to the pixel point to be processed under the first motion estimation vector, and the second calculation window pixel difference corresponding to the pixel point to be processed under the second motion estimation vector; and use the motion estimation vector corresponding to the minimum calculation window pixel difference between the first calculation window pixel difference and the second calculation window pixel difference as the target motion estimation vector corresponding to the pixel point to be processed.

[0224] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or parts included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0225] It should be noted that, in the embodiment of the present application, if the above-mentioned dense optical flow generation method is implemented in the form of a software function part and is sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0226] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0227] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0228] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0229] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0230] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0231] Figure 11 is a schematic diagram of the hardware entity of a computer device provided in an embodiment of the present application. As shown in Figure 11, the hardware entity of the computer device 1100 includes: a processor 1101 and a memory 1102, wherein the memory 1102 stores a computer program that can be run on the processor 1101, and when the processor 1101 executes the program, the steps in the method of any of the above embodiments are implemented.

[0232] The memory 1102 stores computer programs that can be run on the processor. The memory 1102 is configured to store instructions and applications executable by the processor 1101. It can also cache data to be processed or processed by the processor 1101 and various parts of the computer device 1100 (for example, image data, audio data, voice communication data, and video communication data). It can be implemented through flash memory (FLASH) or random access memory (RAM).

[0233] When the processor 1101 executes the program, the processor 1101 implements the steps of any of the above-mentioned dense optical flow generation methods. The processor 1101 generally controls the overall operation of the computer device 1100.

[0234] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the dense optical flow generation method in any of the above embodiments.

[0235] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0236] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, which are not specifically limited in the embodiments of the present application.

[0237] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0238] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0239] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0241] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0242] In addition, the functional units in the embodiments of the present application can all be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units. It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the above-mentioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), magnetic disks or optical disks.

[0243] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional part and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words be embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0244] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application. Industrial Applicability

[0245] The embodiment of the present application discloses a dense optical flow generation method, apparatus, device and storage medium, wherein the method includes: performing block matching on the current image frame and the next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; the first motion estimation vector corresponding to the pixel to be processed is determined by the block matching result corresponding to the unit image block where the pixel to be processed is located; based on the matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame, the second motion estimation vector corresponding to each pixel to be processed is fitted; the matching pixel is determined based on the pixel to be processed and the corresponding first motion estimation vector; based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed, dense optical flow information is generated. In the embodiment of the present application, the first motion estimation vector of the pixel to be processed at the image block level and the second motion estimation vector of the pixel to be processed at the pixel level are obtained respectively, and the final dense optical flow information is generated based on the first motion estimation vector and the second motion estimation vector. In this way, the obtained dense optical flow information not only focuses on the global information of the image but also the local information of the image, thereby improving the accuracy of the dense optical flow information; at the same time, by using the first motion estimation vector at the image block level to determine the preliminary direction during the pixel-level motion estimation process, compared with the dense optical flow calculation scheme of the related technology, since it is not necessary to consider all the image information of the image frame, multiple iteration processes are eliminated, and the optical flow calculation time is reduced.

Claims

1. A dense optical flow generation method, the method comprising: Performing block matching on the current image frame and the next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; The first motion estimation vector corresponding to the pixel to be processed is determined by a block matching result corresponding to the unit image block where the pixel to be processed is located; Fitting a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a local image of each pixel to be processed in the current image frame and a local image of a corresponding matching pixel in the next image frame; The matching pixel point is determined based on the pixel point to be processed and the corresponding first motion estimation vector; Based on the first motion estimation vector and the second motion estimation vector corresponding to each of the to-be-processed pixels, dense optical flow information is generated.

2. The method according to claim 1, wherein: The first motion estimation vector corresponding to the pixel to be processed is a target block motion vector of the unit image block where the pixel to be processed is located; The performing block matching on the current image frame and the next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame includes: For each preset image block in the current image frame, determining an image block pixel difference of the preset image block at different division scales; Based on the image block pixel differences of the preset image block at different division scales, a target block motion vector of each unit image block in the preset image block is determined.

3. The method according to claim 2, wherein: The different division scales include at least a first division scale corresponding to the preset image block, and determining the target block motion vector of each unit image block in the preset image block based on the image block pixel difference of the preset image block at different division scales includes: In response to the image block pixel difference at the first division scale being greater than or equal to the image block pixel difference at any other division scale, for each of the unit image blocks, based on the block motion vector corresponding to the unit image block at each of the division scales, a target block motion vector of the unit image block is determined.

4. The method according to claim 2, wherein: The different division scales include at least a first division scale corresponding to the preset image block, and determining the target block motion vector of each unit image block in the preset image block based on the image block pixel difference of the preset image block at different division scales includes: In response to the image block pixel difference at the first division scale being smaller than the image block pixel difference at each other division scale, the block motion vector corresponding to the preset image block is determined as the target block motion vector of each unit image block in the preset image block.

5. The method according to claim 2, wherein: The determining of the pixel difference of the preset image block at different division scales includes: For each of the division scales, an image block pixel difference between a first sub-image block and a corresponding second sub-image block under the division scale is determined respectively; the first sub-image block is obtained by dividing the preset image block based on the division scale; the second sub-image block is an image block in the next image frame with the smallest image block pixel difference with the first sub-image block; The sum of the image block pixel differences corresponding to the first sub-image block at the division scale is used as the image block pixel difference of the preset image block at the division scale.

6. The method according to claim 3, wherein: The determining, based on the block motion vector corresponding to the unit image block at each of the division scales, the target block motion vector of the unit image block comprises: Based on the block motion vector corresponding to the unit image block at each of the division scales, generating the image block pixel difference corresponding to the unit image block at each of the division scales; The block motion vector corresponding to the minimum image block pixel difference is determined as the target block motion vector of the unit image block.

7. The method according to claim 3 or 4, wherein: The method further comprises: For each first sub-image block at each of the division scales, determining a search area corresponding to the first sub-image block in the next image frame; the first sub-image block is obtained by dividing the preset image block based on the division scale; In the search area corresponding to the first sub-image block, intercepting the image blocks to be selected with a sliding window corresponding to the size of the first sub-image block, and calculating the image block pixel difference between each of the image blocks to be selected and the first sub-image block; Determine the to-be-selected image block corresponding to the smallest image block pixel difference as the second sub-image block corresponding to the first sub-image block; Based on the first sub-image block and the corresponding second sub-image block, a block motion vector corresponding to the first sub-image block at the division scale is generated; wherein the block motion vector corresponding to the unit image block within the first sub-image block at the division scale is the same as the block motion vector corresponding to the first sub-image block.

8. The method according to any one of claims 1 to 6, wherein: The fitting of the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame includes: For each pixel to be processed, a plurality of neighboring pixel points corresponding to the pixel to be processed are obtained in the current image frame, and a plurality of neighboring pixel points corresponding to the matching pixel point are obtained in the next image frame; Based on the matching relationship between a plurality of neighboring pixel points corresponding to the pixel point to be processed and a plurality of neighboring pixel points corresponding to the matching pixel point, a second motion estimation vector corresponding to each pixel point to be processed is fitted.

9. The method according to claim 8, wherein: The step of fitting a second motion estimation vector corresponding to each pixel to be processed based on a matching relationship between a plurality of neighboring pixel points corresponding to the pixel to be processed and a plurality of neighboring pixel points corresponding to the matching pixel point comprises: Based on a plurality of neighboring pixel points corresponding to the pixel point to be processed, a first matrix function is constructed; the first matrix function includes a first pixel function of each neighboring pixel point corresponding to the pixel point to be processed, and the first pixel function is a quadratic polynomial function that characterizes the correlation between the position coordinates of the neighboring pixel points in the current image frame and the grayscale values ​​of the neighboring pixel points; Based on a plurality of neighborhood pixel points corresponding to the neighborhood pixel point, a second matrix function is constructed; the second matrix function includes a second pixel function of each neighborhood pixel point corresponding to the neighborhood pixel point, and the second pixel function is a quadratic polynomial function that characterizes the correlation between the position coordinates of the neighborhood pixel point in the next image frame and the grayscale value of the neighborhood pixel point; Based on the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function, a second motion estimation vector corresponding to the pixel point to be processed is generated.

10. The method according to claim 9, wherein: The quadratic polynomial coefficients of the pixel function include matrix coefficients and vector coefficients; the generating the second motion estimation vector corresponding to the pixel to be processed based on the matching relationship between the first pixel function determined by the first matrix function and the second pixel function determined by the second matrix function includes: Determine a mean matrix coefficient based on a first matrix coefficient corresponding to the first pixel function and a second matrix coefficient corresponding to the second pixel function; Determine a difference vector coefficient based on a first vector coefficient corresponding to the first pixel function and a second vector coefficient corresponding to the second pixel function; The product of the inverse matrix corresponding to the mean matrix coefficient, the preset coefficient and the difference vector coefficient is determined as the second motion estimation vector corresponding to the pixel to be processed.

11. The method according to any one of claims 1 to 6, wherein: The dense optical flow information includes a target motion estimation vector corresponding to each pixel to be processed; The generating dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel to be processed includes: For each of the to-be-processed pixels, based on a preset pixel difference calculation window, respectively determine a first calculation window pixel difference corresponding to the to-be-processed pixel under the first motion estimation vector, and a second calculation window pixel difference corresponding to the to-be-processed pixel under the second motion estimation vector; The motion estimation vector corresponding to the minimum calculation window pixel difference between the first calculation window pixel difference and the second calculation window pixel difference is used as the target motion estimation vector corresponding to the pixel point to be processed.

12. A dense optical flow generating device, the device comprising: A first vector acquisition part is configured to perform block matching on a current image frame and a next image frame to obtain a first motion estimation vector corresponding to each pixel to be processed in the current image frame; The first motion estimation vector corresponding to the pixel to be processed is determined by a block matching result corresponding to the unit image block where the pixel to be processed is located; The second vector acquisition part is configured to fit the second motion estimation vector corresponding to each pixel to be processed based on the matching relationship between the local image of each pixel to be processed in the current image frame and the local image of the corresponding matching pixel in the next image frame; The matching pixel point is determined based on the pixel point to be processed and the corresponding first motion estimation vector; The dense optical flow generating part is configured to generate dense optical flow information based on the first motion estimation vector and the second motion estimation vector corresponding to each pixel point to be processed.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 11 when executing the program.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising computer readable codes, wherein when the computer readable codes are executed in an electronic device, a processor in the electronic device executes the steps for implementing the method according to any one of claims 1 to 11.

Images