Optical flow calculation methods, terminal devices and storage media

Abstract

This application relates to an optical flow calculation method, a terminal device, and a storage medium. The terminal device performs texture mapping on each of two adjacent image frames to obtain an image pyramid, and determines an image layer of the image pyramid as the initial calculation layer based on the image resolution of the frame image. The terminal device determines an image layer of the image pyramid as the final calculation layer based on a preset output optical flow resolution, and determines multiple calculation layers. The terminal device also calculates the output optical flow value of the initial calculation layer based on the initial calculation layer of the two image frames using a preset optical flow algorithm. The output optical flow value of the initial calculation layer is used as the input data for the next image layer, and the preset optical flow algorithm is used to calculate the output optical flow value of the next calculation layer until the output optical flow value of the final calculation layer is obtained. This application can adaptively calculate the optical flow of input frame images with different resolutions according to the preset optical flow algorithm, and avoids excessive load on the optical flow algorithm.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an optical flow calculation method, terminal device, and storage medium. Background Technology

[0002] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a moving object in space. Optical flow methods utilize the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Optical flow can be considered an instantaneous velocity, and when the time interval is very small (such as between two consecutive frames in a video), it can be used as the displacement of a target point. For example, due to the movement of the subject or camera, a target point in the image moves between two consecutive frames; this movement can be called optical flow. Optical flow reflects the instantaneous velocity of a moving object in the two-dimensional space of the image, reflecting the magnitude and direction of the object's motion. Therefore, optical flow calculation is one of the prerequisites for motion estimation. Optical flow calculation methods can be divided into dense optical flow and sparse optical flow. Each type of optical flow algorithm has its own advantages and disadvantages. Different algorithms with different characteristics are selected for different application scenarios, and the algorithm model and parameters need to be redesigned according to the specific input / output scenario and overall load requirements.

[0003] Due to the complex and varied input and output scenarios of mobile terminals, and the stringent requirements on algorithm load, fixed optical flow algorithms struggle to cope with the dynamic changes in these scenarios. For example, when running a game on a mobile terminal, users can change settings such as image resolution and frame rate in real time, causing the input scenario for optical flow calculations to constantly change. When the input image resolution increases, while a fixed optical flow calculation method maintains the invariance of output optical flow accuracy, the increased computational load leads to a higher overall algorithm load. Conversely, when the input image resolution decreases, the resolution requirement for the output optical flow calculation result can be reduced accordingly, but a fixed optical flow calculation method cannot adjust the output resolution. The same principle applies to changes in optical flow calculation accuracy requirements caused by variations in frame rate and other conditions. Therefore, fixed optical flow calculation methods in related technologies cannot make corresponding adjustments to achieve optimal algorithm performance in scenarios where image resolution, frame rate, and other parameters change. Summary of the Invention

[0004] In view of the above, it is necessary to provide an optical flow calculation method, terminal device and storage medium to solve the technical problem that the existing fixed optical flow calculation method cannot make corresponding adjustments in scenarios where the image resolution changes.

[0005] In a first aspect, embodiments of this application provide an optical flow calculation method, the method comprising: acquiring two adjacent frames of images in an image sequence; performing texture mapping on each frame of images to obtain an image pyramid for each frame of images, wherein the image pyramid includes multiple image layers; acquiring image parameter information for each frame of images, determining an image layer of the image pyramid as an initial calculation layer based on the image parameter information, the image parameters including image resolution; acquiring a preset output optical flow resolution, and determining an image layer of the image pyramid as a final calculation layer based on the preset output optical flow resolution; determining multiple calculation layers from the image pyramid based on the initial calculation layer and the final calculation layer; sequentially calculating the output optical flow values ​​of the multiple calculation layers using a preset optical flow algorithm until an optical flow calculation result is obtained, comprising: calculating the output optical flow value of the initial calculation layer of the two frames of images based on the preset optical flow algorithm; using the output optical flow value of the initial calculation layer of the two frames of images as input data for the next image layer of the initial calculation layer of the two frames of images, calculating the output optical flow value of the next calculation layer using the preset optical flow algorithm until the output optical flow value of the final calculation layer is obtained, and using the output optical flow value of the final calculation layer as the optical flow calculation result. The above technical solution can adaptively calculate the optical flow of input frame images with different image resolutions based on the preset optical flow algorithm without switching the algorithm, ensuring that the optical flow algorithm does not generate a large load when calculating the optical flow of images with different resolutions.

[0006] In one embodiment of this application, the preset optical flow algorithm includes: calculating a first optical flow value using a first preset optical flow algorithm based on an initial calculation layer of two frames of images; obtaining a second optical flow value of the initial calculation layer of the two frames of images through iterative calculation using a second preset optical flow algorithm based on the first optical flow value; and using the second optical flow value as the output optical flow value of the initial calculation layer of the two frames of images. The above technical solution, by fusing the first and second preset optical flow algorithms to calculate optical flow, can dynamically adjust the parameters of the two optical flow algorithms and their degree of participation to achieve better overall performance.

[0007] In one embodiment of this application, the first preset optical flow algorithm is a block-matching optical flow algorithm, and the second preset optical flow algorithm is a Lucas-Kanada sparse optical flow algorithm. The above technical solution, by fusing the block-matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to calculate optical flow, can dynamically adjust the parameters of the block-matching optical flow algorithm and the degree of their participation to achieve better overall performance.

[0008] In one embodiment of this application, the two frames include a reference frame image and a current frame image. Based on the initial calculation layers of the two frames, a first preset optical flow value is calculated using a first preset optical flow algorithm. This includes: dividing the initial calculation layers of the reference frame image and the current frame image into multiple non-overlapping pixel blocks according to the size of preset blocks; for each pixel block of the initial calculation layer of the current frame image, according to a preset matching criterion, performing matching operations on all pixel blocks within the search range of the initial calculation layer of the reference frame image and each pixel block of the initial calculation layer of the current frame image, determining the pixel block with the smallest matching error with each pixel block of the initial calculation layer of the current frame image as the matching pixel block; calculating the optimal motion vector estimate based on the relative position between each pixel block of the initial calculation layer of the current frame image and the corresponding matching pixel block; and determining the first optical flow value based on all optimal motion vector estimates. The above technical solution performs layered processing of the frame image according to an image pyramid and uses a block matching optical flow algorithm to calculate the optical flow of the frame image, thus optimizing the problem that the block matching optical flow algorithm is prone to mismatches in areas of large displacement and motion occlusion.

[0009] In one embodiment of this application, the method further includes: determining a search range based on the image frame rate of a reference frame image. The above technical solution, without switching algorithms, can adaptively calculate the optical flow of input frame images with different image frame rates based on a preset optical flow algorithm, ensuring that the optical flow algorithm does not incur a large load when calculating the optical flow of images with different image frame rates.

[0010] In one embodiment of this application, determining the search range based on the image frame rate of the reference frame image includes: if the image frame rate of the reference frame image is greater than a preset image frame rate or greater than the image frame rate of the adjacent previous frame image of the reference frame image, reducing the search range of pixel blocks in the block matching optical flow algorithm and reducing the number of iterations of the Lucas-Kanada sparse optical flow algorithm; if the image frame rate of the reference frame image is less than the preset image frame rate or less than the image frame rate of the adjacent previous frame image of the reference frame image, increasing the search range of pixel blocks in the matching optical flow algorithm and increasing the number of iterations of the Lucas-Kanada sparse optical flow algorithm. In the above technical solution, if the image frame rate of the reference frame image is greater than the preset image frame rate or greater than the image frame rate of the previous frame image of the reference frame image, it indicates that the accuracy requirement for optical flow calculation is lower. The terminal device reduces the search range of pixel blocks in the block matching optical flow algorithm and reduces the number of iterations of the Lucas-Kanada sparse optical flow algorithm, thereby accelerating the calculation and reducing the overall load of the algorithm. If the frame rate of the reference frame image is less than the preset frame rate or less than the frame rate of the previous frame image of the reference frame image, the terminal device increases the search range of pixel blocks in the matching optical flow algorithm and increases the number of iterations of the Lucas-Kanada sparse optical flow algorithm to meet the accuracy requirements for optical flow calculation.

[0011] In one embodiment of this application, a second optical flow value for the initial computation layer of two frames of images is obtained through iterative calculation using a second preset optical flow algorithm. This includes: iteratively calculating all pixels of the initial computation layer of the two frames of images based on a preset pixel window, and adding the result of each iteration to a first optical flow value to obtain the second optical flow value. This technical solution improves the accuracy of optical flow calculation while reducing the computational load of the Lucas-Kanade sparse optical flow algorithm by using the Lucas-Kanade sparse optical flow algorithm to iteratively calculate all pixels of the initial computation layer of the two frames of images.

[0012] In one embodiment of this application, the output optical flow value of the initial computation layer is used as the input data of the next image layer of the initial computation layer. The output optical flow value of the next computation layer is calculated using a preset optical flow algorithm until the output optical flow value of the termination computation layer is obtained. This includes: using the output optical flow value of the initial computation layer as the initial optical flow value of the block matching optical flow algorithm, and calculating the first optical flow value of the next computation layer of two frames of images using the block matching optical flow algorithm; using the first optical flow value of the next computation layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, iteratively calculating the second optical flow value of the next computation layer of two frames of images using the updated Lucas-Kanada sparse optical flow algorithm, using the second optical flow value of the next computation layer as the output optical flow value of the next computation layer, and sequentially performing iterative calculations layer by layer on multiple computation layers until the output optical flow value of the termination computation layer is obtained. The above technical solution achieves optical flow calculation for frame images by fusing the block matching optical flow algorithm with the Lucas-Kanada sparse optical flow algorithm. In the process of calculating optical flow, there is no need to obtain image feature points for optical flow calculation. At the same time, without switching algorithms, it can adaptively calculate the optical flow of input images with different image resolutions, ensuring that the algorithm does not generate a large load.

[0013] In one embodiment of this application, determining an image layer of the image pyramid as the initial calculation layer based on image parameter information includes: determining the corresponding image layer as the initial calculation layer based on the image resolution of each frame image, according to a preset image layer relationship table. The image layer relationship table includes the correspondence between multiple image resolutions and multiple initial calculation layers, with each image resolution corresponding to one initial calculation layer. This technical solution can quickly find the initial calculation layer based on image parameter information using the image layer relationship table, thereby further reducing the algorithm load.

[0014] In one embodiment of this application, determining an image layer of the image pyramid as the termination calculation layer based on a preset output optical flow resolution includes: calculating the output optical flow resolution of each image layer of the image pyramid, and using the image layer corresponding to the output optical flow resolution consistent with the preset output optical flow resolution as the termination calculation layer. By using the image layer corresponding to the output optical flow resolution consistent with the preset output optical flow resolution as the termination calculation layer, the above technical solution ensures that the optical flow algorithm does not incur a large load when calculating the optical flow of images with different resolutions.

[0015] In one embodiment of this application, calculating the output optical flow resolution of each image layer of the image pyramid includes:

[0016] The image resolution of each image layer is obtained, and the ratio of the image resolution of each image layer to a preset block is used as the output optical flow resolution of each image layer. The preset block is a pixel domain with a preset size. This technical solution uses the ratio of the image resolution of each image layer to the preset block as the output optical flow resolution of each image layer, which ensures the accuracy requirements of the optical flow algorithm in calculating the optical flow of images with different resolutions.

[0017] In one embodiment of this application, texture mapping is performed on each frame of image to obtain an image pyramid for each frame of image, including: scaling each frame of image sequentially according to a preset equal ratio to obtain multiple image layers, each image layer being an image obtained by scaling each frame of image according to the equal ratio, and all image layers of each frame of image constituting the image pyramid of each frame of image.

[0018] In one embodiment of this application, determining an image layer of the image pyramid as the initial calculation layer based on image parameter information includes: if the first image resolution in the image parameter information is higher than the second image resolution, the position of the initial calculation layer of the frame image corresponding to the first image resolution in the image pyramid is lower than the position of the initial calculation layer of the frame image corresponding to the second image resolution in the image pyramid. The above technical solution adjusts the initial calculation layer in the image pyramid according to the image resolution, thereby ensuring that the workload and accuracy of the fusion block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm are constant when calculating the optical flow of the initial calculation layer.

[0019] Secondly, embodiments of this application provide a terminal device, which includes a memory and a processor: the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the terminal device performs the above-described optical flow calculation method.

[0020] Thirdly, embodiments of this application provide a computer-readable storage medium storing program instructions that, when executed on a terminal device, cause the terminal device to perform the aforementioned optical flow calculation method.

[0021] Furthermore, the technical effects brought about by the second and third aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A software structure block diagram of a terminal device provided in an embodiment of this application.

[0024] Figure 2 A flowchart of an optical flow calculation method provided in an embodiment of this application.

[0025] Figure 3 This is a schematic diagram of an image pyramid provided in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of an image layer relationship table provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart of a method for sequentially calculating the output optical flow values ​​of multiple computing layers using a preset optical flow algorithm in one embodiment of this application.

[0028] Figure 6 This is a schematic diagram illustrating the fusion calculation of optical flow using the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm provided in an embodiment of this application.

[0029] Figure 7 This is a schematic diagram illustrating the calculation of the first optical flow value between the initial computation layers of two frames in one embodiment of this application.

[0030] Figure 8 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c (seven cases).

[0033] The movement of a target point in an image across two consecutive frames due to the movement of the subject or camera is called optical flow. Optical flow reflects the instantaneous velocity of a moving object in the two-dimensional space of an image, representing the magnitude and direction of the object's velocity. Therefore, optical flow calculation is one of the prerequisites for motion estimation. Optical flow calculation methods can be divided into dense optical flow and sparse optical flow methods. Each type of optical flow algorithm has its own advantages and disadvantages. Different algorithms with different characteristics are selected for different application scenarios, and the algorithm model and parameters need to be redesigned according to the specific input / output scenario and overall load requirements.

[0034] Due to the complex and varied input and output scenarios of mobile terminals, and the stringent requirements on algorithm load, fixed optical flow algorithms struggle to cope with the dynamic changes in these scenarios. For example, when running a game on a mobile terminal, users can change settings such as image resolution and frame rate in real time, causing the input scenario for optical flow calculations to constantly change. When the input image resolution increases, while a fixed optical flow calculation method maintains the invariance of output optical flow accuracy, the increased computational load leads to a higher overall algorithm load. Conversely, when the input image resolution decreases, the resolution requirement for the output optical flow calculation result can be reduced accordingly, but a fixed optical flow calculation method cannot adjust the output resolution. The same principle applies to changes in optical flow calculation accuracy requirements caused by variations in frame rate and other conditions. Therefore, fixed optical flow calculation methods in related technologies cannot make corresponding adjustments to achieve optimal algorithm performance in scenarios where image resolution, frame rate, and other parameters change.

[0035] To address the aforementioned problems, this application provides an optical flow calculation method. The method is applied in a terminal device. The terminal device includes an application processor that runs an operating system. The operating system of the terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses a layered Android system as an example to exemplify the software structure of the terminal device.

[0036] Figure 1 This is a software architecture block diagram of a terminal device provided in one embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into six layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer, the kernel layer, and the hardware layer. The application layer may include a series of application packages.

[0037] like Figure 1 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0038] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0039] like Figure 1As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0040] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0041] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0042] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0043] A phone manager is used to provide communication functions for terminal devices. For example, it manages call status (including connection and disconnection).

[0044] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0045] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0046] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0047] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0048] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0049] System libraries can include multiple functional modules. For example: surfaceflinger, surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0050] The surface painter is a drawing engine for creating graphics.

[0051] The Surface Manager is used to manage the display subsystem and provides the blending of two-dimensional and three-dimensional layers for multiple applications.

[0052] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0053] A 2D graphics engine is a drawing engine for 2D drawing.

[0054] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0055] The Hardware Abstraction Layer (HAL) is the layer between the system library and the kernel layer. The HAL includes at least the graphics processing unit (GPU) and the hardware composer (HWC) module.

[0056] The fingerprint module is used to report fingerprint events to the fingerprint service in the application framework layer.

[0057] The HWC module is a driver abstraction layer for a dedicated chip for layer compositing. For example, in the embodiments of this application, the HWC module is used to connect the surface painter and the display driver. That is, the HWC module is a communication bridge between the surface painter and the display driver, so that the layers drawn by the surface painter are transmitted to the display driver for display.

[0058] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver, touch driver, fingerprint driver, audio driver, and sensor driver.

[0059] The hardware layer includes at least a Display Driver Integrated Circuit (DDIC), an ambient light sensor, a touch sensor, and a display screen. The DDIC sends drive signals and data to the display panel, controlling screen brightness and color to display images, letters, and other visual content. The ambient light sensor detects the intensity or color of the surrounding light. The touch sensor detects user touch operations on the display screen.

[0060] The following will combine Figure 2 The optical flow calculation method of this application is described in detail. The optical flow calculation method can be executed in the hardware abstraction layer or kernel layer of the terminal device. For example, the optical flow calculation method can be executed in the GPU of the hardware abstraction layer of the terminal device. Figure 2 The example method includes one or more steps, but does not constitute a limitation of this application. Furthermore, the order of the steps in the method is merely illustrative and may be changed. Additional steps may be added or steps may be removed without departing from the disclosure of this application. The method specifically includes the following steps.

[0061] Step S201: Obtain two adjacent frames of images from the image sequence.

[0062] In one embodiment of this application, the GPU of the terminal device acquires a set of input frame sequence images and extracts two adjacent frame images from the set of frame sequence images. For example, the GPU acquires a set of frame sequence images captured by a camera and extracts two adjacent frame images from the frame sequence images; or, the GPU acquires a set of frame sequence images from the video storage device of the terminal device and extracts two adjacent frame images from the frame sequence images.

[0063] Step S202: Perform texture mapping (Mipmap) on each frame image to obtain the image pyramid of each frame image, wherein the image pyramid includes multiple image layers.

[0064] In one embodiment of this application, obtaining an image pyramid for each frame image by texture mapping includes: sequentially scaling each frame image according to a preset equal-value ratio to obtain multiple image layers, where each image layer is an image obtained after scaling the frame image according to the equal-value ratio, and all image layers of each frame image constitute the image pyramid of that frame image. In one embodiment of this application, the preset equal-value ratio is 2. For example, refer to... Figure 3The original image of a frame has a pixel size of 256x256. The original image is then scaled sequentially according to a preset equal-ratio to obtain a set of child images with pixel sizes of 128x128, 64x64, 32x32, 16x16, 8x8, 4x4, 2x2, and 1x1. The original image is used as the 0th layer of the image pyramid. The 128x128 child image obtained after the first scaling is used as the 1st layer of the image pyramid. The 64x64 child image obtained after the second scaling is used as the 2nd layer of the image pyramid. The third scaling... The 32x32 pixel sub-image obtained after scaling is used as the third image layer of the image pyramid. The 16x16 pixel sub-image obtained after the fourth scaling is used as the fourth image layer. The 8x8 pixel sub-image obtained after the fifth scaling is used as the fifth image layer. The 4x4 pixel sub-image obtained after the sixth scaling is used as the sixth image layer. The 2x2 pixel sub-image obtained after the seventh scaling is used as the seventh image layer. The 1x1 pixel sub-image obtained after the eighth scaling is used as the eighth image layer. In one embodiment of this application, the fewer pixels an image layer has, the higher its position in the image pyramid.

[0065] Step S203: Obtain image parameter information for each frame of image, and determine an image layer from the image pyramid as the initial calculation layer based on the image parameter information. The image parameter information includes at least the image resolution.

[0066] In one embodiment of this application, the terminal device inputs the image pyramid of each frame and the image parameter information of each frame into the GPU's computer shader. The computer shader obtains the image resolution of each frame and determines an image layer of the image pyramid as the initial computation layer based on the image resolution.

[0067] In one embodiment of this application, determining an image layer of the image pyramid as the initial computation layer based on image parameter information includes: determining the image layer in the image pyramid corresponding to the image resolution of each frame image as the initial computation layer based on a preset image layer relationship table. In one embodiment of this application, the image layer relationship table includes a correspondence between multiple image resolutions and multiple initial computation layers, with each image resolution corresponding to one initial computation layer. In one embodiment of this application, the computer shader determines the initial computation layer corresponding to the image resolution by looking up the image layer relationship table based on the image resolution of each frame image. (See reference...) Figure 4The diagram shown is a schematic of an image layer relationship table provided in one embodiment of this application. The initial computation layer corresponding to a frame image resolution of 256x256 is the 5th image layer; the initial computation layer corresponding to a frame image resolution of 128x128 is the 4th image layer; and the initial computation layer corresponding to a frame image resolution of 64x64 is the 3rd image layer. In one embodiment of this application, if the first image resolution in the image parameter information is higher than the second image resolution, the position of the initial computation layer of the frame image corresponding to the first image resolution in the image pyramid is higher than the position of the initial computation layer of the frame image corresponding to the second image resolution in the image pyramid. In one embodiment of this application, the higher the resolution of each frame image, the more details the image contains. The computer shader uses the higher-resolution image layer in the image pyramid as the initial computation layer, which ensures the workload and accuracy of calculating optical flow using the initial computation layer.

[0068] Step S204: Obtain the preset output optical flow resolution, and determine one image layer of the image pyramid as the termination calculation layer based on the preset output optical flow resolution. The output optical flow resolution can be determined based on the magnitude of the output optical flow.

[0069] In one embodiment of this application, the output optical flow resolution is a preset constant value, which can be set according to the user's needs. In another embodiment, the computer shader calculates the output optical flow resolution of each image layer of the image pyramid, and uses the image layer with an output optical flow resolution consistent with the preset resolution as the termination layer. In one embodiment, calculating the output optical flow resolution of each image layer of the image pyramid includes: obtaining the image resolution of each image layer, and using the ratio of the image resolution of each image layer to a preset block as the output optical flow resolution of each image layer. In one embodiment, the preset block is a pixel domain of a preset size, which can be set as needed, for example, to an 8x8 pixel domain.

[0070] Step S205: Determine multiple computation layers from the image pyramid based on the initial computation layer and the final computation layer.

[0071] In one embodiment of this application, the computer shader uses all image layers between the initial computation layer and the final computation layer in the image pyramid as computation layers. For example, refer to... Figure 3 If the computer shader determines that the 4th image layer in the image pyramid is the initial computation layer and the 1st image layer is the final computation layer, then the 4th, 3rd, 2nd, and 1st image layers in the image pyramid are determined as computation layers.

[0072] Step S206: The output optical flow values ​​of multiple computing layers are calculated sequentially using a preset optical flow algorithm until the optical flow calculation result is obtained.

[0073] refer to Figure 5 The diagram shown is a flowchart illustrating a method for sequentially calculating the output optical flow values ​​of multiple computational layers using a preset optical flow algorithm, according to an embodiment of this application. The method specifically includes the following steps.

[0074] Step S501: Based on the initial calculation layer of the two frames of images, calculate the output optical flow value of the initial calculation layer using a preset optical flow algorithm.

[0075] In one embodiment of this application, the preset optical flow algorithm includes: calculating a first optical flow value using a first preset optical flow algorithm based on an initial calculation layer of two frames of images; obtaining a second optical flow value of the initial calculation layer of two frames of images through iterative calculation using a second preset optical flow algorithm based on the first optical flow value; and using the second optical flow value as the output optical flow value of the initial calculation layer of two frames of images.

[0076] Step S502: The output optical flow value of the initial calculation layer is used as the input data of the next image layer of the initial calculation layer. The output optical flow value of the next calculation layer is calculated using a preset optical flow algorithm until the output optical flow value of the termination calculation layer is obtained. The output optical flow value of the termination calculation layer is then used as the optical flow calculation result.

[0077] The above technical solution can adaptively calculate the optical flow of input frame images with different image resolutions based on the preset optical flow algorithm without switching the algorithm, ensuring that the optical flow algorithm does not generate a large load when calculating the optical flow of images with different resolutions.

[0078] In one embodiment of this application, the first preset optical flow algorithm is a block-matching optical flow algorithm, and the second preset optical flow algorithm is the Lucas-Kanada sparse optical flow algorithm. In another embodiment of this application, the two image frames include a current frame image and a reference frame image. Figure 6The diagram illustrates the fusion calculation of optical flow using a block-matching optical flow algorithm and a Lucas-Kanada sparse optical flow algorithm according to an embodiment of this application. In one embodiment, based on the initial computation layers of two frames, a first preset optical flow algorithm is used to calculate a first optical flow value. This includes: dividing the initial computation layer of the reference frame image and the initial computation layer of the current frame image into multiple non-overlapping pixel blocks according to a preset block size; for each pixel block of the initial computation layer of the current frame image, according to a preset matching criterion, performing matching operations on all pixel blocks within the search range of the initial computation layer of the reference frame image and each pixel block of the initial computation layer of the current frame image, determining the pixel block with the smallest matching error to each pixel block of the initial computation layer of the current frame image as the matching pixel block; calculating the optimal motion vector estimate based on the relative position between each pixel block of the initial computation layer of the current frame image and the corresponding matching pixel block; and determining the first optical flow value based on all optimal motion vector estimates. In one embodiment of this application, the size of the preset block can be set to an 8x8 pixel size.

[0079] In one embodiment of this application, the search range can be determined based on the image frame rate of a reference frame image. In another embodiment, a search range relationship table is consulted to determine the search range corresponding to the image frame rate of the frame image. The search range relationship table includes multiple relationships between search ranges and multiple image frame rates, with each image frame rate corresponding to one search range.

[0080] In one embodiment of this application, if the image frame rate of the reference frame image is greater than a preset image frame rate or greater than the image frame rate of the adjacent previous frame image of the reference frame image, the search range of pixel blocks in the block matching optical flow algorithm is reduced according to the image frame rate of the reference frame image, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is reduced; if the image frame rate of the reference frame image is less than a preset image frame rate or less than the image frame rate of the previous frame image of the reference frame image, the search range of pixel blocks in the matching optical flow algorithm is increased according to the image frame rate of the reference frame image, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is increased. For example, if the frame rate of the reference frame image is 30fps, and the frame rate of the adjacent previous frame image is 25fps, the pixel block of the matching optical flow algorithm corresponding to the previous frame image is an 8*8 pixel domain, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is 8, then based on the frame rate of the reference frame image, the search range of the pixel block in the matching optical flow algorithm is reduced from an 8*8 pixel domain to a 4*4 pixel domain, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is reduced from 8 to 6. In one embodiment of this application, the size of the search range of the matching optical flow algorithm and the number of iterations of the Lucas-Kanada sparse optical flow algorithm can be determined by looking up the corresponding parameter relationship table based on the frame rate of the reference frame image. The parameter relationship table includes the correspondence between multiple different image frame rates, the size of the search range of multiple matching optical flow algorithms, and the number of iterations of multiple Lucas-Kanada sparse optical flow algorithms.

[0081] refer to Figure 7 The diagram shown is a schematic representation of calculating the first optical flow value between the initial computation layers of two frames of images in one embodiment of this application. Figure 7As shown, image F1 is the initial calculation layer for the current frame image, and image F2 is the initial calculation layer for the reference frame image. Image F1 is divided into non-overlapping pixel blocks of size M*N pixels according to a preset block size. Given a search range (um, vm), all pixel blocks in image F1 are matched according to a preset matching criterion. For example, for a pixel block with center coordinates (x, y) in image F1, the search range for the corresponding pixel block in image F2 is the region from (x-um, y-vm) to (x+um, y+vm). All pixel blocks within the search range of image F2 are matched with the pixel blocks in image F1 to obtain the pixel block with the minimum matching error. The difference between the starting point coordinates of the pixel block with the minimum matching error and the starting point coordinates (x, y) of the pixel block in image F1 is used as the optimal motion vector estimate of the pixel block in image F1, where um, vm represents the number of pixels, and (x, y) represents the pixel coordinates. In one embodiment of this application, the preset matching criterion is the Sum of Absolute Differences algorithm.

[0082] In one embodiment of this application, the second optical flow value of the initial calculation layer of two frames of images is obtained by iterative calculation using a second preset optical flow algorithm, including: performing multiple iterative calculations on all pixels of the initial calculation layer of the two frames of images based on a preset pixel window, and adding the result of each iteration to the first optical flow value to obtain the second optical flow value. In one embodiment of this application, the Lucas-Kanada sparse optical flow algorithm, based on the assumptions of constant brightness (i.e., assuming that the brightness or gray value of a pixel remains constant during motion), temporal persistence (i.e., assuming that the movement of an object in the image changes slowly over time), and spatial consistency (i.e., assuming that adjacent pixels have similar motion), obtains the constraint equation Ix*u + Iy*v + It = 0 for the image, where Ix, Iy, and It represent the partial derivatives of the gray value of the pixel in the image layer along the x, y, and t directions, (x, y) represents the pixel coordinates, t represents time, and (u, v) represents the optical flow vector. The constraint equation Ix*u + Iy*v + It = 0 is iterated multiple times based on a pixel window to obtain multiple iterations of the optical flow. Each iteration result of the optical flow is added to the first optical flow value to obtain the second optical flow value, or the last iteration result of the optical flow is added to the first optical flow value to obtain the second optical flow value.

[0083] In one embodiment of this application, the computer shader uses the output optical flow value of the initial computation layer as the input data of the next image layer of the initial computation layer, and calculates the output optical flow value of the next computation layer using a preset optical flow algorithm until the output optical flow value of the terminating computation layer is obtained. This includes: using the output optical flow value of the initial computation layer as the initial optical flow value of the block matching optical flow algorithm, calculating the first optical flow value of the next computation layer of two frames of images using the block matching optical flow algorithm, using the first optical flow value of the next computation layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm to update the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, iteratively calculating the second optical flow value of the next computation layer of two frames of images using the updated Lucas-Kanada sparse optical flow algorithm, using the second optical flow value of the next computation layer as the output optical flow value of the next computation layer, and sequentially performing iterative calculations layer by layer on multiple computation layers until the output optical flow value of the terminating computation layer is obtained.

[0084] For example, the third image layer of the image pyramid is the initial computation layer, and the first image layer is the final computation layer. After calculating the output optical flow value of the initial computation layer (i.e., the third image layer), this output optical flow value is used as the initial optical flow value for the block-matching optical flow algorithm. The first optical flow value of the second image layer is calculated using the block-matching optical flow algorithm, and this first optical flow value is used as the initial optical flow value for the Lucas-Kanada sparse optical flow algorithm to update the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm. Finally, the second optical flow value of the second image layer of two frames is iteratively calculated using the Lucas-Kanada sparse optical flow algorithm. The second optical flow value of the second image layer is used as the initial optical flow value for the block-matching optical flow algorithm to update the initial optical flow value of the block-matching optical flow algorithm. Based on the updated block-matching optical flow algorithm, the first optical flow value of the first image layer is calculated. This first optical flow value is then used as the initial optical flow value for the Lucas-Kanada sparse optical flow algorithm to update its initial optical flow value. Based on the updated Lucas-Kanada sparse optical flow algorithm, the second optical flow value of the first image layer for both frames is calculated to obtain the output optical flow value of the first image layer. This output optical flow value is then used as the output optical flow value of the final calculation layer. (Reference) Figure 5 After calculating the output optical flow value for each image layer, the output can be performed.

[0085] In one embodiment of this application, the output optical flow value of the termination calculation layer can be used to identify actions in a frame image, or for video tracking or video frame interpolation. For example, in one embodiment of this application, the frame image to be identified and the output optical flow value are respectively input into the ResNeXt3D subnet of the video action recognition model to determine the image features of the frame image and the optical flow features of the output optical flow value. The image features and optical flow features are respectively input into the bidirectional attention layer of the video action recognition model to determine the first weighted feature after weighting the image features and the second weighted feature after weighting the optical flow features. The first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the user action corresponding to the user contained in the frame image to be identified.

[0086] In related technologies, the block matching optical flow algorithm can optimize the problem of mismatches in large displacement and motion-occluded regions by using image pyramid layering when calculating optical flow. Therefore, the block matching optical flow algorithm is suitable for large motion scenes. However, the computational accuracy of the block matching optical flow algorithm is low (pixel level) and it is prone to getting trapped in local optimum matching problems. In contrast, the Lucas-Kanade sparse optical flow algorithm derives local information from a small window (i.e., a pixel window) around the set of image feature points to obtain the optical flow result of the feature point set. Therefore, the Lucas-Kanade sparse optical flow algorithm is suitable for small motion scenes and has high computational accuracy (sub-pixel level). However, it requires obtaining image feature point information, increasing the computational load of the algorithm. In the above embodiments of this application, an image layer of the image pyramid is determined as the initial calculation layer based on the image resolution, and the final calculation layer of the image pyramid is determined based on the resolution of the output optical flow. For the initial calculation layer of the image pyramid, the first optical flow value between two frames is calculated using a block-matching optical flow algorithm as a rough optical flow estimate. This optical flow estimate is then used as the initial optical flow value for the Lucas-Kanade sparse optical flow algorithm, and the second optical flow value is calculated using the Lucas-Kanade sparse optical flow algorithm. Finally, the output optical flow value of the initial calculation layer is obtained by combining the first optical flow value with the second optical flow value. The output optical flow value is used as the input data for the next image layer of the initial calculation layer. The output optical flow value of each image layer is calculated layer by layer until the output optical flow value of the final calculation layer is obtained, which is used as the optical flow calculation result. In this way, by fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, the optical flow of the frame image can be calculated. In the process of calculating the optical flow, there is no need to obtain image feature points for optical flow calculation. At the same time, without switching the optical flow algorithm, it can adaptively perform optical flow calculation for input frame images with different image resolutions, ensuring that the optical flow algorithm does not generate a large load.

[0087] In one embodiment of this application, the pixel block size of the block-matching optical flow algorithm and the pixel window size of the Lucas-Kanada sparse optical flow algorithm are the same; for example, both the pixel block size and the pixel window size are 8*8 pixel domains. Thus, steps S501 and S502 can be processed in parallel in the GPU's computer shader in a workgroup or thread-based manner. The number of workgroups or threads can be the ratio of the image layer resolution to the pixel block size.

[0088] In one embodiment of this application, the size of the search block (i.e., pixel block) and the search path of the block-matching optical flow algorithm affect the algorithm's performance. The number of iterations and other termination conditions of the Lucas-Kanada sparse optical flow algorithm affect its performance. In this embodiment, the computer shader uses the fused block-matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm for optical flow calculation. The fused processing of the block-matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm includes: dynamically adjusting the parameters of the block-matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, as well as their degree of participation, according to the image resolution, the image frame rate, the matching error, or the output optical flow resolution to obtain better overall performance, thereby further realizing an adaptive algorithm for calculating optical flow in different scenes.

[0089] In one embodiment of this application, the computer shader determines an image layer of the image pyramid as the initial computation layer based on the image resolution. A higher resolution input frame image indicates more detail within the frame image. If the image resolution is high, the computer shader adjusts the lower-positioned image layer in the image pyramid as the initial computation layer based on the image resolution; conversely, if the image resolution is low, the computer shader adjusts the lower-positioned image layer in the image pyramid as the initial computation layer. This ensures that the workload and accuracy of the fusion block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm are consistent when calculating the optical flow of the initial computation layer.

[0090] In one embodiment of this application, the computer shader performs fusion processing on each image layer from the initial computation layer to the final computation layer using a block-matching optical flow algorithm and a Lucas-Kanada sparse optical flow algorithm to obtain the output optical flow value of the corresponding image layer. The more image layers in the image pyramid between the initial and final computation layers, the greater the algorithm's load. To balance the algorithm's load, when the number of image layers between the initial and final computation layers exceeds a preset number, the computer shader designates the (N-1)th image layer below the initial computation layer in the image pyramid as the target image layer, where N is a preset number (e.g., 4). Each image layer from the initial computation layer to the target image layer is processed using the block-matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to obtain the corresponding output optical flow value. The output optical flow value of the target image layer is then passed layer by layer to the next image layer until it reaches the final image layer. During the process of passing the output optical flow value of the target image layer, it is not necessary to recalculate the output optical flow value of the next layer; adjustments are only needed based on a preset multiplier. For example, in one embodiment, when the output optical flow value of the target image layer is passed between each image layer, the output optical flow value of the target image layer is multiplied by a preset multiple, where the preset multiple is the size ratio of two adjacent image layers. For example, the preset multiple can be set to 2, but this is only an example and is not limited to this in actual applications.

[0091] In one embodiment of this application, a smaller frame interval of the input frame image indicates a higher image frame rate and less motion between the two frames. In another embodiment, if the frame interval between the two input frames decreases / the image frame rate increases, it indicates a lower requirement for optical flow calculation accuracy. The computer shader reduces the search range of pixel blocks in the block-matching optical flow algorithm and reduces the number of iterations in the Lucas-Kanada sparse optical flow algorithm, thereby accelerating computation and reducing the overall algorithm load. Conversely, if the frame interval increases / the image frame rate decreases, the requirement for optical flow calculation accuracy increases. The computer shader increases the search range of pixel blocks in the matching optical flow algorithm and increases the number of iterations in the Lucas-Kanada sparse optical flow algorithm to meet the accuracy requirements for optical flow calculation.

[0092] In one embodiment of this application, during the calculation of the output optical flow of each image layer, the computershader determines whether to execute the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow based on the magnitude of the minimum matching error found by the block matching optical flow algorithm. In one embodiment of this application, determining whether to execute the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow based on the magnitude of the minimum matching error found by the block matching optical flow algorithm includes: if the minimum matching error is within a first preset error range, the Lucas-Kanada sparse optical flow algorithm is not executed; if the minimum matching error is within a second preset error range, when executing the Lucas-Kanada sparse optical flow algorithm, the number of iterations for calculating the optical flow is reduced, or a preset error threshold is used as an early termination condition for the iteration of the Lucas-Kanada sparse optical flow algorithm. The first preset error range and the second preset error range can be set according to user needs, wherein the first preset error range is smaller than the second preset error range.

[0093] In one embodiment of this application, when the computer shader determines that the current image layer of the image pyramid has received the output optical flow value transmitted from the previous image layer, it first filters the received output optical flow value, then determines the initial matching error of the current image layer based on the filtered output optical flow value, and adjusts the search path and search range of the block matching optical flow method based on the initial matching error, so as to reduce the initial matching error and enable the matching operation of the block matching optical flow algorithm to be completed quickly, thereby improving the computational efficiency.

[0094] The terminal device 100 involved in the embodiments of this application is described below.

[0095] refer to Figure 8 The diagram shown illustrates the hardware structure of a terminal device 100 according to an embodiment of this application. The terminal device 100 may include a processor 110, internal memory 130, external memory interface 120, display screen 140, and sensor module 150. The sensor module 150 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0096] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0097] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0098] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0099] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0100] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0101] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0102] The terminal device 100 implements display functions through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0103] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 140, where N is a positive integer greater than 1.

[0104] Internal memory 130 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM may include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, generally referred to as DDR5 SDRAM), etc. The NVM may include disk storage devices and flash memory. Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0105] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0106] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0107] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0108] Internal memory 130 or external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 110. The one or more computer programs include multiple instructions, which, when executed by processor 110, can implement the optical flow calculation method executed on terminal device 100 in the above embodiments, so as to realize the optical flow calculation function of terminal device 100.

[0109] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the optical flow calculation method described in the above embodiment.

[0110] In addition, some embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the optical flow calculation method in the above-described method embodiments.

[0111] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of some embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application should not depart from the spirit and scope of the technical solutions of this application.

Claims

1. A method for calculating optical flow, characterized in that, The method includes: Get two adjacent frames from an image sequence; Texture mapping is performed on each frame of the image to obtain the image pyramid of each frame, wherein the image pyramid includes multiple image layers; The image parameter information of each frame is obtained, and an image layer of the image pyramid is determined as the initial calculation layer based on the image parameter information. The image parameters include the image resolution. Obtain a preset output optical flow resolution, and determine an image layer of the image pyramid as the termination calculation layer based on the preset output optical flow resolution; Based on the initial calculation layer and the termination calculation layer, multiple calculation layers are determined from the image pyramid; The output optical flow values ​​of the multiple computation layers are calculated sequentially using a preset optical flow algorithm until the optical flow calculation result is obtained, including: Based on the initial calculation layer of the two frames of images, the output optical flow value of the initial calculation layer of the two frames of images is calculated using the preset optical flow algorithm; The output optical flow value of the initial computation layer of the two frames is used as the input data of the next computation layer of the initial computation layer of the two frames. The output optical flow value of the next computation layer is calculated using the preset optical flow algorithm until the output optical flow value of the termination computation layer is obtained. The output optical flow value of the termination computation layer is used as the optical flow calculation result. The preset optical flow algorithm includes: calculating a first optical flow value based on the initial computation layer of the two frames using a first preset optical flow algorithm; obtaining a second optical flow value of the initial computation layer of the two frames using a second preset optical flow algorithm through iterative calculation based on the first optical flow value; and using the second optical flow value as the output optical flow value of the initial computation layer of the two frames.

2. The optical flow calculation method as described in claim 1, characterized in that, The first preset optical flow algorithm is a block matching optical flow algorithm, and the second preset optical flow algorithm is the Lucas-Kanada sparse optical flow algorithm.

3. The optical flow calculation method as described in claim 2, characterized in that, The two frames include a reference frame image and a current frame image. The initial calculation layer based on the two frames calculates a first optical flow value using a first preset optical flow algorithm, including: The initial calculation layer of the reference frame image and the initial calculation layer of the current frame image are divided into multiple non-overlapping pixel blocks according to the size of the preset blocks. For each pixel block of the initial calculation layer of the current frame image, according to the preset matching criteria, all pixel blocks within the search range of the initial calculation layer of the reference frame image are matched with each pixel block of the initial calculation layer of the current frame image, and the pixel block with the smallest matching error with each pixel block of the initial calculation layer of the current frame image is determined as the matching pixel block. Based on the relative position between each pixel block of the initial computation layer of the current frame image and its corresponding matching pixel block, the optimal motion vector estimate is calculated, and the first optical flow value is determined based on all the optimal motion vector estimates.

4. The optical flow calculation method as described in claim 3, characterized in that, The method further includes: The search range is determined based on the image frame rate of the reference frame image.

5. The optical flow calculation method as described in claim 4, characterized in that, Determining the search range based on the image frame rate of the reference frame image includes: If the frame rate of the reference frame image is greater than the preset frame rate or greater than the frame rate of the adjacent previous frame image of the reference frame image, the search range of pixel blocks in the block matching optical flow algorithm is reduced, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is reduced. If the frame rate of the reference frame image is less than the preset frame rate or less than the frame rate of the adjacent previous frame image of the reference frame image, the search range of pixel blocks in the matching optical flow algorithm is increased, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is increased.

6. The optical flow calculation method as described in claim 1, characterized in that, The step of obtaining the second optical flow value of the initial computation layer of the two frames of images through iterative calculation using a second preset optical flow algorithm includes: Based on a preset pixel window, multiple iterations are performed on all pixels of the initial calculation layer of the two frames of images, and the result of each iteration is added to the first optical flow value to obtain the second optical flow value.

7. The optical flow calculation method as described in claim 1, characterized in that, The step of using the output optical flow value of the initial calculation layer as input data for the next image layer, and calculating the output optical flow value of the next calculation layer using the preset optical flow algorithm, until the output optical flow value of the final calculation layer is obtained, includes: The output optical flow value of the initial computation layer is used as the initial optical flow value of the block matching optical flow algorithm, and the first optical flow value of the next computation layer of the two frames of images is calculated by the block matching optical flow algorithm. The first optical flow value of the next computational layer is used as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm. The second optical flow value of the next computational layer of the two frames of images is iteratively calculated through the updated Lucas-Kanada sparse optical flow algorithm. The second optical flow value of the next computational layer is used as the output optical flow value of the next computational layer. The multiple computational layers are iteratively calculated layer by layer until the output optical flow value of the terminating computational layer is obtained.

8. The optical flow calculation method as described in claim 1, characterized in that, The step of determining an image layer of the image pyramid as the initial calculation layer based on the image parameter information includes: Based on a preset image layer relationship table, the corresponding image layer is determined as the initial calculation layer according to the image resolution of each frame image. The image layer relationship table includes the correspondence between multiple image resolutions and multiple initial calculation layers, with each image resolution corresponding to one initial calculation layer.

9. The optical flow calculation method as described in claim 1, characterized in that, The step of determining an image layer of the image pyramid as the termination calculation layer based on a preset output optical flow resolution includes: The output optical flow resolution of each image layer in the image pyramid is calculated, and the image layer corresponding to the output optical flow resolution that is consistent with the preset output optical flow resolution is taken as the termination calculation layer.

10. The optical flow calculation method as described in claim 9, characterized in that, The calculation of the output optical flow resolution for each image layer of the image pyramid includes: The image resolution of each image layer is obtained, and the ratio of the image resolution of each image layer to the preset block is used as the output optical flow resolution of each image layer. The preset block is a pixel field with a preset size.

11. The optical flow calculation method as described in claim 1, characterized in that, The process of performing texture mapping on each frame of an image to obtain the image pyramid for each frame includes: Each frame of the image is scaled sequentially according to a preset equal ratio to obtain the plurality of image layers. Each image layer is an image obtained by scaling each frame of the image according to the equal ratio. All image layers of each frame of the image constitute the image pyramid of each frame of the image.

12. The optical flow calculation method as described in claim 1, characterized in that, The step of determining an image layer of the image pyramid as the initial calculation layer based on the image parameter information includes: If the first image resolution in the image parameter information is higher than the second image resolution, the position of the frame image corresponding to the first image resolution in the initial calculation layer of the image pyramid is higher than the position of the initial calculation layer of the frame image corresponding to the second image resolution in the image pyramid.

13. A terminal device, characterized in that, The terminal device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the terminal device performs the optical flow calculation method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed on a terminal device, cause the terminal device to perform the optical flow calculation method as described in any one of claims 1 to 12.

Images