Image processing method and device based on optical flow estimation and computer device

Abstract

The present disclosure provides an image processing method and device based on optical flow estimation and a storage medium, comprising: acquiring two adjacent images, and respectively performing down-sampling on the pixel points of each image to construct an image pyramid, and dividing each layer image of the image pyramid into N image blocks, wherein N is a positive integer greater than 1, and each image block of adjacent layer images has a corresponding relationship, and calculating first optical flow data according to a first image block in the N image blocks of each layer image, and performing target processing operation on the two images based on the first optical flow data, and calculating second optical flow data according to a second image block in the N image blocks of each layer image, wherein the second optical flow data is used for the target processing operation, and the optical flow data of another part of the region can be calculated while the target processing operation is performed by using part of the optical flow data, so that the overall image processing time can be shortened.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, and storage medium based on optical flow estimation. Background Technology

[0002] In computer vision, optical flow motion estimation algorithms are commonly used to calculate the motion offset of corresponding pixels between two adjacent image frames. They obtain pixel optical flow data, or optical flow offset data, by calculating the offset positions of corresponding pixels between two frames. In calculating this optical flow data, the image can be downsampled to construct a pyramid. First, the optical flow data of the smaller pyramid layer is calculated, and then fused to the optical flow offset of the previous pyramid layer. This fusion process is repeated from top to bottom to obtain the final optical flow data. Further, based on this optical flow data, subsequent image processing such as image stabilization, 3D noise reduction, target segmentation, recognition, tracking, robot navigation, and shape information restoration can be performed.

[0003] In image stabilization and noise reduction processes, related technologies require calculating complete optical flow data from two adjacent images. Subsequent stabilization and noise reduction are performed only after obtaining the complete optical flow data. Therefore, optical flow calculation and subsequent image processing are sequential, and the required computation time is the sum of the computation times of the two steps. Consequently, in high-speed computing scenarios, there will be a certain time delay, which is detrimental to real-time image processing. Summary of the Invention

[0004] This disclosure proposes an image processing method, apparatus, and storage medium based on optical flow estimation, aiming to at least partially solve one of the technical problems in the related art.

[0005] The first aspect of this disclosure proposes an image processing method based on optical flow estimation, comprising: acquiring two adjacent frames of images and downsampling the pixels of each frame of images to construct an image pyramid; dividing each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers of images has a corresponding relationship; calculating first optical flow data based on the first image block among the N image blocks of each layer of images; performing target processing operations on the two frames of images based on the first optical flow data, and simultaneously calculating second optical flow data based on the second image block among the N image blocks of each layer of images, wherein the second optical flow data is used for the target processing operation.

[0006] A second aspect of this disclosure provides an image processing apparatus based on optical flow estimation, comprising: a construction module for acquiring two adjacent frames of images and downsampling the pixels of each frame of images to construct an image pyramid; a block division module for dividing each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers of images has a corresponding relationship; a calculation module for calculating first optical flow data based on a first image block among the N image blocks of each layer of images; and a processing module for performing target processing operations on the two frames of images based on the first optical flow data, and simultaneously calculating second optical flow data based on a second image block among the N image blocks of each layer of images, wherein the second optical flow data is used for the target processing operation.

[0007] A third aspect of this disclosure provides a weighted buffer, comprising: a static random access memory and a latch.

[0008] A fourth aspect of this disclosure provides a computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an image processing method based on optical flow estimation according to an embodiment of this disclosure.

[0009] A fifth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform an image processing method based on optical flow estimation disclosed in embodiments of this disclosure.

[0010] In this embodiment, two adjacent image frames are acquired, and the pixels of each frame are downsampled to construct an image pyramid. Each layer of the image pyramid is divided into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship. Based on the first image block among the N image blocks of each layer, first optical flow data is calculated, and target processing operations are performed on the two images based on the first optical flow data. Simultaneously, based on the second image block among the N image blocks of each layer, second optical flow data is calculated. The second optical flow data is used for the target processing operation and can calculate the optical flow data of a portion of the two images. By using part of the optical flow data for target processing operations while calculating the optical flow data of another portion of the image, the technical effect of parallel optical flow calculation and processing operations is achieved. This can shorten the overall image processing time, reduce latency, and thus meet the needs of real-time image processing.

[0011] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a schematic flowchart of an image processing method based on optical flow estimation according to an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of the structure of an image pyramid according to an embodiment of this disclosure;

[0015] Figure 3A This is a schematic diagram of an arrangement structure of multiple image blocks in each layer of an image pyramid according to an embodiment of the present disclosure;

[0016] Figure 3B This is a schematic diagram of another arrangement structure of multiple image blocks in each layer of the image pyramid provided in the embodiments of this disclosure;

[0017] Figure 4 This is a schematic flowchart of an image processing method based on optical flow estimation according to another embodiment of the present disclosure;

[0018] Figure 5 This is a schematic diagram of an image processing apparatus based on optical flow estimation according to another embodiment of the present disclosure;

[0019] Figure 6 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0020] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0021] It should be noted that the execution subject of the image processing method based on optical flow estimation in this embodiment can be an image processing device based on optical flow estimation. This device can be implemented by software and / or hardware and can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0022] Figure 1 This is a schematic flowchart of an image processing method based on optical flow estimation according to an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes:

[0023] S101: Acquire two adjacent frames of images and downsample the pixels of each frame to construct an image pyramid.

[0024] In this embodiment, the two frames can be two consecutive frames acquired in any image processing scenario. For example, during image stabilization and noise reduction, this embodiment can acquire two adjacent frames, without limitation. It is understood that the two frames in this embodiment have the same size and resolution.

[0025] In this embodiment, the two frames of images acquired can be high-resolution images, that is, images with a large number of pixels. In order to accurately calculate optical flow data, this embodiment can downsample the pixels of each frame of images to construct an image pyramid.

[0026] Figure 2 This is a schematic diagram of the structure of an image pyramid provided according to an embodiment of this disclosure, such as... Figure 2 As shown, this embodiment constructs an image pyramid of m layers (e.g., 3 layers) for each frame image. The first layer is the original image of each frame, whose width and height (number of pixels) can be represented by w1 and h1, respectively. The width and height of the second layer image are represented by w2*h2, ..., and the m-th layer is the top layer, whose width and height (number of pixels) can be represented by w1*h1, respectively. m and h m express.

[0027] In some embodiments, the width and height of the m-1 layer image are, for example, twice that of the m layer image; similarly, the width and height of the m-2 layer image are twice that of the m-1 layer image, and so on. The width and height of the first layer image are twice that of the second layer image, that is to say, w p =2*w p+1 ,h p =2*h p+1 Where p represents the layer number of the image pyramid, belonging to 1 to m-1.

[0028] S102: Divide each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship.

[0029] In other words, in this embodiment, the first, second, ..., m-th layer of the image pyramid are divided into N image blocks. For example, if the pyramid is divided into N image blocks on average, then each image block is 1 / N of the original image layer. In this embodiment, each image block of an adjacent layer has a corresponding relationship; that is, each image block of the m-th layer has a corresponding image block in the (m-1)-th layer, and each image block of the (m-1)-th layer has a corresponding image block in the (m-2)-th layer. In other words, each image block of each layer corresponds to another image.

[0030] Some embodiments, Figure 3A This is a schematic diagram of an arrangement structure of multiple image blocks in each layer of an image pyramid according to an embodiment of this disclosure, such as... Figure 3A As shown, in this embodiment, each layer of the image pyramid can be divided into N image blocks (block1, block2, ..., blockN) with the same width as the current layer image. That is, the N image blocks of the first layer image are equal in width to the first layer image; specifically, each image block of the first layer image has a width of w1 and a height of... Similarly, the N image blocks of the second layer image are equal in width to the width of the second layer image itself; that is, the width of each image block in the second layer image is w2, and the height is... ..., the N image blocks of the m-th layer image are equal in width to the width of the m-th layer image itself, that is: the width of each image block of the m-th layer image is w. m The height is In this context, each image block in each layer of the image has a corresponding relationship, that is: block1 of the first layer image to the m-th layer image has a corresponding relationship, block2 has a corresponding relationship, ..., blockN has a corresponding relationship.

[0031] Other embodiments, Figure 3B This is a schematic diagram illustrating another arrangement structure of multiple image blocks in each layer of the image pyramid provided in the embodiments of this disclosure, such as... Figure 3B As shown, in this embodiment, each layer of the image pyramid can be divided into N image blocks (block1, block2, ..., blockN) with the same height as the current layer image. That is, the N image blocks of the first layer image are equal in height to the first layer image itself; specifically, each image block of the first layer image has a height of h1 and a width of h2. Similarly, the N image blocks of the second layer image are equal in height to the height of the second layer image, that is: the height of each image block of the second layer image is h2, and the width is... ..., the N image blocks of the m-th layer image have the same height as the m-th layer image, that is: the height of each image block of the m-th layer image is h. m Width is

[0032] It is understood that the above embodiments are merely illustrative of dividing N image blocks. In practical applications, there may be other possible division methods, which are not limited thereto.

[0033] S103: Calculate the first optical flow data based on the first image block among the N image blocks of each layer of the image.

[0034] In each layer of the image, any one of the N image blocks can be used as the first image block. For example, block1 can be used as the first image block and block2 can be used as the first image block. There are no restrictions on this.

[0035] The optical flow data (i.e., optical flow offset) calculated based on the pixels in the first image block (i.e., m first image blocks) of each layer of the image can be called the first optical flow data. That is, the first optical flow data is the optical flow data of the first image block region of the first layer of the image (the original image of each frame).

[0036] Specifically, in this embodiment of the present disclosure, a first image block with a corresponding relationship is first determined for each layer of the image. The first image block is, for example, block1 of each layer of the image. That is, m blocks1 from the first layer of the image to the mth layer of the image are determined. Further, the first optical flow data is calculated based on the m blocks1.

[0037] In some embodiments, the optical flow data of the first image block of each layer is calculated sequentially from the top layer (m-th layer image) to the bottom layer (first layer image), and the optical flow data of the first image block of the bottom layer image is used as the first optical flow data. Specifically, this embodiment first calculates the optical flow data component of block1 of the m-th layer image, and fuses this optical flow data component with the optical flow data component of block1 of the previous image layer (i.e., m-1 layer image). The fused optical flow data component is used as the output of the m-1 layer image, and then fused with the optical flow data component of block1 of the m-2 layer image. This fusion is iteratively repeated until the total optical flow data of the first image block block1 of the first layer image is output, and this total optical flow data is used as the first optical flow data. This embodiment does not impose specific limitations on the process of calculating the optical flow data.

[0038] In practical applications, when there are a large number of image blocks, in order to balance the real-time image processing speed and the number of calculations, the embodiments of this disclosure can determine multiple image blocks from N image blocks as first image blocks, and calculate the first optical flow data based on the first image blocks. For example, block1 and block2 can be used together as the first image block, that is, in this embodiment, the optical flow data of the regions of block1 and block2 are calculated as the first optical flow data.

[0039] S104: Perform target processing operation on two frames of images based on the first optical flow data, and calculate the second optical flow data based on the second image block in the N image blocks of each layer of the image. The second optical flow data is used for the target processing operation.

[0040] After calculating the first optical flow data as described above, the embodiments of this disclosure can first perform target processing operations on the two frames of images based on the first optical flow data. That is to say, target processing operations can be performed on the first image block regions of the two frames of images. The target processing operations can be, for example, image stabilization, noise reduction, target tracking, and any other possible processing, without limitation.

[0041] Furthermore, in this embodiment, while performing target processing operations on two frames of images based on the first optical flow data, the second optical flow data can be calculated based on the second image block among the N image blocks of each layer of the image.

[0042] In this system, any one of the N image blocks in each image layer, excluding the first image block, can be used as a second image block, such as block2. The optical flow data calculated for the second image block block2 of each image layer can be called the second optical flow data. This second optical flow data is used for target processing operations, such as image stabilization and noise reduction for the second image block region of two frames. The calculation method for the second optical flow data is the same as that for the first optical flow data, and will not be elaborated here.

[0043] In other words, the embodiments of this disclosure first calculate the first optical flow data of the previous image block (first image block), and while performing target processing operations on two frames of images using the first optical flow data, calculate the second optical flow data of other image blocks (second image blocks), and further perform target processing operations on two frames of images using the second optical flow data, and so on. While calculating the optical flow data of a part of the image, the optical flow data of another part of the image is used to process the image, so that the optical flow calculation and processing operations can be performed in parallel.

[0044] In practical applications, assuming the time required to calculate the optical flow at layer m of the entire frame is Tm, the time required to calculate the optical flow at layer (m-1) of the entire frame is Tm-1, and so on, the time required to calculate the optical flow at layer 2 of the entire frame is T2. The output result of the optical flow at layer 1 is the final optical flow information we need. The optical flow information at layers 2 to m is the intermediate result. The optical flow calculation at layer (m-1) requires the result of the optical flow calculation at layer m, and the optical flow calculation at layer 1 requires the output result of the optical flow at layer 2. According to the traditional optical flow calculation method, the time required to obtain the result of the optical flow at layer 1 is T... a=Tm + Tm-1 + ... + T1; It can be understood that target processing operations on an image also involve processing multiple image block regions sequentially. Assume the time for target processing operations on the entire frame is T. b The total time for image processing is T. a +T b Compared to traditional methods, this scheme calculates the optical flow data for each image block, thus reducing the time required for each image block to [missing information]. Similarly, the time required to perform target processing on each image block region is... In this embodiment, while performing target processing using the optical flow data of one image block, the optical flow data of other image blocks are calculated simultaneously. Therefore, the target processing time and the optical flow calculation time overlap, i.e.: and There will be overlap, so the time does not need to be accumulated. Therefore, the total time spent on image processing can be reduced.

[0045] In this embodiment, two adjacent image frames are acquired, and the pixels of each frame are downsampled to construct an image pyramid. Each layer of the image pyramid is divided into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship. Based on the first image block among the N image blocks of each layer, first optical flow data is calculated, and target processing operations are performed on the two images based on the first optical flow data. Simultaneously, based on the second image block among the N image blocks of each layer, second optical flow data is calculated. The second optical flow data is used for the target processing operation and can calculate the optical flow data of a portion of the two images. By using part of the optical flow data for target processing operations while calculating the optical flow data of another portion of the image, the technical effect of parallel optical flow calculation and processing operations is achieved. This can shorten the overall image processing time, reduce latency, and thus meet the needs of real-time image processing.

[0046] Figure 4 This is a flowchart illustrating an image processing method based on optical flow estimation according to another embodiment of the present disclosure, as shown below. Figure 4 As shown, the method includes:

[0047] S401: Acquire two adjacent frames of images and downsample the pixels of each frame to construct an image pyramid.

[0048] For a detailed description of S401, please refer to the above embodiments, which will not be repeated here.

[0049] S402: Determine the number of N image blocks to be divided based on the number of pixels in each frame.

[0050] In other words, this embodiment can determine the specific value of N based on the number of pixels in the width and height of each frame image. For example, if the number of pixels in each frame image is large, the number of image blocks can be increased; if the number of pixels in each frame image is small, the number of image blocks can be reduced.

[0051] In some embodiments, a relationship table between the number of image blocks (N) and the number of pixels can be pre-configured. After acquiring two frames of images, the number of image blocks corresponding to the number of pixels in the image can be queried based on the relationship table. Alternatively, the number of image blocks corresponding to the number of pixels in each frame of the image can be calculated according to a preset formula. For example, the integer part obtained by dividing the number of pixels by a set value is used as the number of image blocks N. There is no limitation on this.

[0052] In other embodiments, the image block division method can be determined based on the number of pixels in width and height of each frame (i.e., w and h). For example, if the number of pixels in width is greater than the number of pixels in height (w is greater than h), then a vertical division method is used (i.e., Figure 3B If the number of wide pixels is less than the number of high pixels (w < h), then a vertical division method is used (i.e., Figure 3A Therefore, image blocks can be divided according to the number of pixels in the image, thus allowing for flexible processing of images of different sizes and making it more widely applicable.

[0053] S403: Divide each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of an adjacent layer has a corresponding relationship.

[0054] S404: Calculate the first optical flow data based on the first image block among the N image blocks of each layer of the image.

[0055] For detailed explanations of S403-S404, please refer to the above embodiments, which will not be repeated here.

[0056] S405: Write the first optical flow data into memory.

[0057] In this embodiment of the disclosure, after calculating the first optical flow data, the first optical flow data can be written into memory (Double Data Rate, DDR).

[0058] S406: Read the first optical flow data from memory and perform target processing operations on the two frames of images based on the first optical flow data.

[0059] Furthermore, during target processing, the first optical flow data is first read from DDR, and then target processing is performed on the two frames of images based on the first optical flow data. In this embodiment, because the first optical flow data is read from memory, the data reading speed can be improved, further enhancing image processing efficiency.

[0060] In this embodiment, two adjacent image frames are acquired, and the pixels of each frame are downsampled to construct an image pyramid. Each layer of the image pyramid is divided into N image blocks, where N is a positive integer greater than 1, and each image block in an adjacent layer has a corresponding relationship. First optical flow data is calculated based on the first image block from the N image blocks in each layer, and target processing operations are performed on the two images based on the first optical flow data. Simultaneously, second optical flow data is calculated based on the second image block from the N image blocks in each layer. This second optical flow data is used for the target processing operation and can calculate the optical flow data of a portion of the two images. By using partial optical flow data for target processing while simultaneously calculating the optical flow data of another portion, the technical effect of parallel optical flow calculation and processing operations is achieved. This shortens the overall image processing time, reduces latency, and meets the needs of real-time image processing. Furthermore, this embodiment can divide the image blocks according to the number of pixels in the image, thus allowing for flexible processing of images of different sizes and a wider range of applications. Furthermore, this embodiment reads the first optical flow data from memory, thus improving the data reading speed and further enhancing image processing efficiency.

[0061] Figure 5 This is a schematic diagram of an image processing apparatus based on optical flow estimation according to another embodiment of the present disclosure. Figure 5 As shown, the image processing apparatus 50 based on optical flow estimation includes:

[0062] The construction module 501 is used to acquire two adjacent frames of images and downsample the pixels of each frame of images to construct an image pyramid.

[0063] The segmentation module 502 is used to divide each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship.

[0064] The calculation module 503 is used to calculate the first optical flow data based on the first image block among the N image blocks of each layer of image;

[0065] The processing module 504 is used to perform target processing operations on two frames of images based on the first optical flow data, and at the same time calculates the second optical flow data based on the second image block in the N image blocks of each layer of the image, wherein the second optical flow data is used for the target processing operation.

[0066] In some embodiments, the device 50 further includes: a storage module for writing the first optical flow data into memory; and a processing module 504 specifically for: reading the first optical flow data from memory and performing target processing operations on the two frames of images based on the first optical flow data.

[0067] In some embodiments, the apparatus further includes a determining module for determining the number of N image blocks to be divided based on the number of pixels in each frame of the image.

[0068] In some embodiments, the segmentation module 502 is specifically used to: divide each layer of the image pyramid into N image blocks with the same width as the current layer image; or divide each layer of the image pyramid into N image blocks with the same height as the current layer image.

[0069] In some embodiments, the calculation module 503 is specifically used to: calculate the optical flow data of the first image block of each layer of image in the order from the top layer to the bottom layer, and use the optical flow data of the first image block of the bottom layer image as the first optical flow data.

[0070] In some embodiments, the calculation module 503 is specifically used to: determine multiple image blocks among N image blocks of each layer image as first image blocks; and calculate first optical flow data based on the first image blocks.

[0071] In this embodiment, two adjacent image frames are acquired, and the pixels of each frame are downsampled to construct an image pyramid. Each layer of the image pyramid is divided into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship. Based on the first image block among the N image blocks of each layer, first optical flow data is calculated, and target processing operations are performed on the two images based on the first optical flow data. Simultaneously, based on the second image block among the N image blocks of each layer, second optical flow data is calculated. The second optical flow data is used for target processing operations and can calculate the optical flow data of a portion of the two images. By using part of the optical flow data for target processing operations while simultaneously calculating the optical flow data of another portion of the image, the technical effect of parallel optical flow calculation and processing operations is achieved. This can shorten image processing time, reduce latency, and thus meet the needs of real-time image processing.

[0072] According to embodiments of this disclosure, this disclosure also provides a computer device, a readable storage medium, and a computer program product.

[0073] To implement the above embodiments, this disclosure also proposes a computer program product that, when executed by an instruction processor, performs an image processing method based on optical flow estimation as proposed in the foregoing embodiments of this disclosure.

[0074] Figure 6 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 6The computer device 12 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0075] like Figure 6 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0076] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0077] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0078] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 6 Not shown; usually referred to as a "hard drive".

[0079] although Figure 6Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0080] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0081] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0082] The processing unit 16 executes various functional applications by running programs stored in the system memory 28, such as implementing the image processing method based on optical flow estimation mentioned in the foregoing embodiments.

[0083] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0084] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0085] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0086] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0087] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0088] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0089] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0090] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0091] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An image processing method based on optical flow estimation, characterized in that, The method includes: Two adjacent frames of images are acquired, and the pixels of each frame are downsampled to construct an image pyramid. Each layer of the image pyramid is divided into N image blocks, where N is a positive integer greater than 1, and each image block of an adjacent layer has a corresponding relationship. Calculate the first optical flow data based on the first image block among the N image blocks of each layer of the image; Based on the first optical flow data, target processing operations are performed on the regions corresponding to the first image blocks in the two frames of images. At the same time, second optical flow data is calculated based on the second image blocks among the N image blocks of each layer of images. The second optical flow data is used to perform target processing operations on the regions corresponding to the second image blocks in the two frames of images. The first image block region and the second image block region are non-overlapping block regions.

2. The method of claim 1, wherein, After calculating the first optical flow data, the method further includes: Write the first optical flow data into memory; Furthermore, the target processing operation based on the first optical flow data for the two frames of images includes: The first optical flow data is read from the memory, and target processing operations are performed on the two frames of images based on the first optical flow data.

3. The method of claim 1, wherein, Before dividing each layer of the image pyramid into N image blocks, the process includes: The number of N image blocks is determined based on the number of pixels in each frame.

4. The method of claim 1, wherein, The step of dividing each layer of the image pyramid into N image blocks includes: Each layer of the image pyramid is divided into N image blocks with the same width as the current layer; or Each layer of the image pyramid is divided into N image blocks with the same height as the current layer.

5. The method of claim 1, wherein, The step of calculating the first optical flow data based on the first image block among the N image blocks of each image layer includes: The optical flow data of the first image block of each layer of the image is calculated sequentially from the top layer to the bottom layer, and the optical flow data of the first image block of the bottom layer image is used as the first optical flow data.

6. The method of claim 1, wherein, The step of calculating the first optical flow data based on the first image block among the N image blocks of each image layer includes: Determine multiple image blocks from N image blocks in each image layer as the first image block; and The first optical flow data is calculated based on the first image block.

7. An image processing apparatus based on optical flow estimation, characterized by, include: The building module is used to acquire two adjacent frames of images and downsample the pixels of each frame to build an image pyramid. The segmentation module is used to divide each layer of the image pyramid into N image blocks, where N is a positive integer greater than 1, and each image block of adjacent layers has a corresponding relationship. The calculation module is used to calculate the first optical flow data based on the first image block among the N image blocks of each layer of the image; The processing module is used to perform target processing operations on the region corresponding to the first image block in the two frames of images based on the first optical flow data, and at the same time calculate the second optical flow data based on the second image block among the N image blocks of each layer of images. The second optical flow data is used to perform target processing operations on the region corresponding to the second image block in the two frames of images. The first image block region and the second image block region are non-overlapping block regions.

8. The apparatus of claim 7, wherein, The device further includes: Storage module, used to write the first optical flow data into memory; Furthermore, the processing module is specifically used to: read the first optical flow data from the memory, and perform target processing operations on the two frames of images based on the first optical flow data.

9. The apparatus of claim 7, wherein, The device further includes: The determining module is used to determine the number of N image blocks to be divided based on the number of pixels in each frame of the image.

10. The apparatus of claim 7, wherein, The segmentation module is specifically used for: Each layer of the image pyramid is divided into N image blocks with the same width as the current layer; or Each layer of the image pyramid is divided into N image blocks with the same height as the current layer.

11. The apparatus of claim 7, wherein, The computing module is specifically used for: The optical flow data of the first image block of each layer of the image is calculated sequentially from the top layer to the bottom layer, and the optical flow data of the first image block of the bottom layer image is used as the first optical flow data.

12. The apparatus of claim 7, wherein, The computing module is specifically used for: Determine multiple image blocks from N image blocks in each image layer as the first image block; and The first optical flow data is calculated based on the first image block.

13. A computer device, comprising: Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-6.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images