Optical flow calculation method, terminal device and storage medium

Abstract

The embodiment of the invention relates to an optical flow calculation method, terminal equipment and a storage medium. The terminal device performs texture mapping on each frame of image in two adjacent frames of images to obtain an image pyramid, and determines one image layer of the image pyramid as an initial calculation layer according to the image resolution of the frame of image. And the terminal equipment determines one image layer of the image pyramid as a termination calculation layer according to a preset output optical flow resolution, and determines a plurality of calculation layers. The terminal device also calculates an output optical flow value of the initial calculation layer by using a preset optical flow algorithm based on the initial calculation layer of the two frames of images, uses the output optical flow value of the initial calculation layer as input data of a next image layer of the initial calculation layer, and calculates an output optical flow value of the next calculation layer by using the preset optical flow algorithm. And the output optical flow value of the termination calculation layer is obtained. According to the invention, the optical flow of the input frame images with different resolutions can be calculated in a self-adaptive manner according to the preset optical flow algorithm, and an overlarge load caused by the optical flow algorithm is avoided.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly relates to an optical flow calculation method, a terminal device, and a storage medium. Background Art

[0002] Optical flow is the instantaneous velocity of the pixels of a moving object in space on the observation imaging plane. The optical flow method is a method that uses the change of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, so as to calculate the motion information of the object between adjacent frames. Optical flow can be regarded as an instantaneous rate, and when the time interval is very small (such as between two consecutive frames of a video), it can be used as the displacement of the target point. For example, due to the movement of the shooting object or the camera, the target point in the image moves between two consecutive frames, and this movement can be called optical flow. Optical flow reflects the instantaneous velocity of a moving object in the two-dimensional space of the image, and reflects the magnitude and direction of the motion velocity of the moving object. Therefore, optical flow calculation is one of the prerequisites for realizing motion estimation. Optical flow calculation methods can be divided into dense optical flow methods (Dense Flow) and sparse optical flow methods (Sparse Flow). Each type of optical flow algorithm has its own advantages and disadvantages. For different application scenarios, algorithms with different characteristics will be selected, and the model, parameters, etc. of the algorithm need to be redesigned according to the specific input and output scenarios and overall load requirements.

[0003] Due to the complex and ever-changing input and output scenarios of mobile terminals, and at the same time, the requirements for algorithm load are extremely strict, fixed optical flow algorithms are difficult to cope with the dynamic changes of the scenarios of mobile terminals. For example, taking a mobile terminal running a game application as an example, users can change settings such as the resolution and frame rate of the game application image in real time, resulting in continuous changes in the input scenario of optical flow calculation on the mobile terminal. When the resolution of the input image becomes larger, although the fixed optical flow calculation method maintains the invariance of the output optical flow accuracy, the overall load of the algorithm increases due to the increase in the amount of calculation; on the contrary, when the resolution of the input image becomes smaller, the resolution requirement for the optical flow calculation result on the output side can be correspondingly reduced, but the fixed optical flow calculation method cannot adjust the output resolution of the optical flow. The same is true for changes in the accuracy requirements of optical flow calculation caused by changes in other conditions such as frame rate. Therefore, the fixed optical flow calculation methods in related technologies cannot make corresponding adjustments in scenarios where the image resolution, frame rate, etc. change to achieve the best effect of the algorithm. Summary of the Invention

[0004] In view of the above, it is necessary to provide an optical flow calculation method, a terminal device, and a storage medium to solve the technical problem that the existing fixed optical flow calculation method cannot make corresponding adjustments in scenarios where the image resolution changes.

[0005] In a first aspect, an embodiment of the present application provides an optical flow calculation method, and the method includes: obtaining two adjacent frames of images in an image sequence; performing texture mapping on each frame of image to obtain an image pyramid for each frame of image, where the image pyramid includes multiple image layers; obtaining image parameter information of each frame of image, and determining an image layer of the image pyramid as an initial calculation layer according to the image parameter information, and the image parameters include image resolution; obtaining a preset output optical flow resolution, and determining an image layer of the image pyramid as a termination calculation layer according to the preset output optical flow resolution; determining multiple calculation layers from the image pyramid according to the initial calculation layer and the termination calculation layer; calculating output optical flow values of the multiple calculation layers in sequence through a preset optical flow algorithm until an optical flow calculation result is obtained, including: based on the initial calculation layer of the two frames of images, calculating the output optical flow value of the initial calculation layer of the two frames of images by using the preset optical flow algorithm; using the output optical flow value of the initial calculation layer of the two frames of images as input data for the next image layer of the initial calculation layer of the two frames of images, and using the preset optical flow algorithm to calculate the output optical flow value of the next calculation layer until the output optical flow value of the termination calculation layer is obtained, and using the output optical flow value of the termination calculation layer as the optical flow calculation result. The above technical solution can adaptively calculate the optical flow of input frame images with different image resolutions according to the preset optical flow algorithm without switching the algorithm, ensuring that the optical flow algorithm will not generate a large load when calculating the optical flow of images with different resolutions.

[0006] In an embodiment of the present application, the preset optical flow algorithm includes: based on the initial calculation layer of the two frames of images, calculating a first optical flow value by using a first preset optical flow algorithm; based on the first optical flow value, obtaining a second optical flow value of the initial calculation layer of the two frames of images through iterative calculation by using a second preset optical flow algorithm, and using the second optical flow value as the output optical flow value of the initial calculation layer of the two frames of images. The above technical solution can perform fusion calculation of the optical flow through the first preset optical flow algorithm and the second preset optical flow algorithm, and can dynamically adjust the parameters of the two optical flow algorithms and their participation degrees to obtain better comprehensive performance.

[0007] In an embodiment of the present application, the first preset optical flow algorithm is a block matching optical flow algorithm, and the second preset optical flow algorithm is a Lucas-Kanada sparse optical flow algorithm. The above technical solution can perform fusion calculation of the optical flow through the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, and can dynamically adjust the parameters of the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm and their participation degrees to obtain better comprehensive performance.

[0008] In an embodiment of the present application, two frames of images include a reference frame image and a current frame image. Based on the initial calculation layers of the two frames of images, a first optical flow value is calculated using a first preset optical flow algorithm, including: dividing the initial calculation layers of the reference frame image and the current frame image into a plurality of non-overlapping pixel blocks according to the size of a preset block; for each pixel block of the initial calculation layer of the current frame image, according to a preset matching criterion, performing a matching operation on all pixel blocks within the search range of the initial calculation layer of the reference frame image and each pixel block of the initial calculation layer of the current frame image, and determining the pixel block with the smallest matching error with each pixel block of the initial calculation layer of the current frame image as the matching pixel block; calculating an optimal motion vector estimation value according to the relative positions between each pixel block of the initial calculation layer of the current frame image and the corresponding matching pixel block, and determining the first optical flow value according to all the optimal motion vector estimation values. The above technical solution performs hierarchical processing on the frame images according to an image pyramid and calculates the optical flow of the frame images using a block matching optical flow algorithm, so as to optimize the problem that the block matching optical flow algorithm is prone to incorrect matching in large displacement and motion occlusion regions.

[0009] In an embodiment of the present application, the method further includes: determining a search range according to the image frame rate of the reference frame image. The above technical solution can adaptively calculate the optical flow of input frame images with different image frame rates according to a preset optical flow algorithm without switching the algorithm, ensuring that the optical flow algorithm does not generate a large load when calculating the optical flow of images with different image frame rates.

[0010] In an embodiment of the present application, determining the search range according to the image frame rate of the reference frame image includes: if the image frame rate of the reference frame image is greater than a preset image frame rate or greater than the image frame rate of the adjacent previous frame image of the reference frame image, reducing the search range of the pixel blocks in the block matching optical flow algorithm and reducing the number of iterations of the Lucas-Kanada sparse optical flow algorithm; if the image frame rate of the reference frame image is less than the preset image frame rate or less than the image frame rate of the adjacent previous frame image of the reference frame image, increasing the search range of the pixel blocks in the matching optical flow algorithm and increasing the number of iterations of the Lucas-Kanada sparse optical flow algorithm. In the above technical solution, if the image frame rate of the reference frame image is greater than the preset image frame rate or greater than the image frame rate of the previous frame image of the reference frame image, it means that the requirement for the calculation accuracy of the optical flow becomes lower. The terminal device reduces the search range of the pixel blocks in the block matching optical flow algorithm and reduces the number of iterations of the Lucas-Kanada sparse optical flow algorithm, thereby accelerating the calculation and reducing the overall load of the algorithm. If the image frame rate of the reference frame image is less than the preset image frame rate or less than the image frame rate of the previous frame image of the reference frame image, the terminal device increases the search range of the pixel blocks in the matching optical flow algorithm and increases the number of iterations of the Lucas-Kanada sparse optical flow algorithm to meet the requirement for the calculation accuracy of the optical flow.

[0011] In an embodiment of the present application, the second optical flow value of the initial calculation layer of two frames of images is obtained by iterative calculation using the second preset optical flow algorithm, including: based on a preset pixel window, performing iterative calculation on all pixels of the initial calculation layer of the two frames of images, and adding the result of each iteration to the first optical flow value to obtain the second optical flow value. In the above technical solution, through the Lucas-Kanade sparse optical flow algorithm, iterative calculation is performed on all pixels of the initial calculation layer of the two frames of images, so as to improve the calculation accuracy of the optical flow while reducing the calculation load of the Lucas-Kanade sparse optical flow algorithm.

[0012] In an embodiment of the present application, the output optical flow value of the initial calculation layer is used as the input data of the next image layer of the initial calculation layer, and a preset optical flow algorithm is used to calculate the output optical flow value of the next calculation layer until the output optical flow value of the termination calculation layer is obtained, including: using the output optical flow value of the initial calculation layer as the initial optical flow value of the block matching optical flow algorithm, and calculating the first optical flow value of the next calculation layer of the two frames of images through the block matching optical flow algorithm; using the first optical flow value of the next calculation layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, and iteratively calculating the second optical flow value of the next calculation layer of the two frames of images through the updated Lucas-Kanada sparse optical flow algorithm, and using the second optical flow value of the next calculation layer as the output optical flow value of the next calculation layer, and performing iterative calculation layer by layer on multiple calculation layers in sequence until the output optical flow value of the termination calculation layer is calculated. In the above technical solution, after fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, the calculation of the optical flow of the frame image is realized. During the calculation of the optical flow, there is no need to obtain image feature points for optical flow calculation, and at the same time, without switching the algorithm, the optical flow of input images with different image resolutions can be adaptively calculated, ensuring that the algorithm will not generate a large load.

[0013] In an embodiment of the present application, determining an image layer of the image pyramid as the initial calculation layer according to the image parameter information includes: based on a preset image layer relationship table, determining the corresponding image layer as the initial calculation layer according to the image resolution of each frame of image. The image layer relationship table includes the corresponding relationship between multiple image resolutions and multiple initial calculation layers, and each image resolution corresponds to an initial calculation layer. In the above technical solution, through the image layer relationship table, the initial calculation layer can be quickly found according to the image parameter information, thereby further reducing the algorithm load.

[0014] In an embodiment of the present application, determining an image layer of an image pyramid as a termination calculation layer according to a preset output optical flow resolution includes: calculating the output optical flow resolution of each image layer of the image pyramid, and using the image layer corresponding to the output optical flow resolution that is consistent with the preset output optical flow resolution as the termination calculation layer. The above technical solution uses the image layer corresponding to the output optical flow resolution that is consistent with the preset output optical flow resolution as the termination calculation layer, which can ensure that the optical flow algorithm does not generate a large load when calculating the optical flow of images with different resolutions.

[0015] In an embodiment of the present application, calculating the output optical flow resolution of each image layer of the image pyramid includes:

[0016] Obtaining the image resolution of each image layer, and using the ratio of the image resolution of each image layer to a preset block as the output optical flow resolution of each image layer, where the preset block is a pixel domain with a preset size. The above technical solution uses the ratio of the image resolution of each image layer to the preset block as the output optical flow resolution of each image layer, which can ensure the accuracy requirements of the optical flow algorithm when calculating the optical flow of images with different resolutions.

[0017] In an embodiment of the present application, performing texture mapping on each frame of image to obtain the image pyramid of each frame of image includes: sequentially scaling each frame of image according to a preset equal ratio to obtain multiple image layers, where each image layer is an image obtained by scaling each frame of image according to the equal ratio, and all the image layers of each frame of image constitute the image pyramid of each frame of image.

[0018] In an embodiment of the present application, determining an image layer of an image pyramid as an initial calculation layer according to image parameter information includes: if the first image resolution in the image parameter information is higher than the second image resolution, the position of the frame image corresponding to the first image resolution in the initial calculation layer of the image pyramid is lower than the position of the frame image corresponding to the second image resolution in the initial calculation layer of the image pyramid. The above technical solution adjusts the initial calculation layer in the image pyramid according to the image resolution, so as to ensure that the workload and accuracy of the fusion block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm are certain when calculating the optical flow of the initial calculation layer.

[0019] In a second aspect, an embodiment of the present application provides a terminal device, which includes a memory and a processor: the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the terminal device executes the above optical flow calculation method.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing program instructions, which, when run on a terminal device, cause the terminal device to execute the above-mentioned optical flow calculation method.

[0021] In addition, for the technical effects brought by the second aspect to the third aspect, reference may be made to the descriptions related to the methods of each design in the above method section, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 It is a software structure block diagram of a terminal device provided by an embodiment of the present application.

[0024] Figure 2 It is a flowchart of an optical flow calculation method provided by an embodiment of the present application.

[0025] Figure 3 It is a schematic diagram of an image pyramid provided by an embodiment of the present application.

[0026] Figure 4 It is a schematic diagram of an image layer relationship table provided by an embodiment of the present application.

[0027] Figure 5 It is a flowchart of a method for sequentially calculating the output optical flow values of multiple calculation layers by a preset optical flow algorithm in an embodiment of the present application.

[0028] Figure 6 It is a schematic diagram of fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow in an embodiment of the present application.

[0029] Figure 7 It is a schematic diagram of calculating the first optical flow value between the initial calculation layers of two frames of images in an embodiment of the present application.

[0030] Figure 8 It is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] Hereinafter, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to mean being an example, illustration or explanation. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that unless otherwise specified in this application, " / " means "or". For example, A / B may mean A or B. The "and / or" in this application is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, these three situations. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c may mean: a, b, c, a and b, a and c, b and c, a, b and c, these seven situations.

[0033] Due to the movement of the object to be photographed or the camera, the movement of the target point in the image between two consecutive frames is called optical flow. Optical flow reflects the instantaneous velocity of a moving object in the two-dimensional image space, reflecting the magnitude and direction of the movement speed of the moving object. Therefore, optical flow calculation is one of the prerequisites for the realization of motion estimation. Optical flow calculation methods can be divided into dense optical flow method (Dense Flow) and sparse optical flow method (Sparse Flow). The advantages and disadvantages of various optical flow algorithms are different. For different application scenarios, algorithms with different characteristics will be selected, and the model, parameters, etc. of the algorithm need to be redesigned according to the specific input and output scenarios and overall load requirements.

[0034] Due to the complex and ever-changing input and output scenarios of mobile terminals, and the stringent requirements for algorithm load, fixed optical flow algorithms are difficult to cope with the dynamic changes in the scenarios of mobile terminals. For example, taking a mobile terminal running a game application as an example, users can change settings such as the resolution and frame rate of the game application image in real time, resulting in continuous changes in the input scenario of the optical flow calculation of the mobile terminal. When the resolution of the input image becomes larger, although the fixed optical flow calculation method maintains the invariance of the output optical flow accuracy, the overall load of the algorithm increases due to the increase in the amount of calculation; conversely, when the resolution of the input image becomes smaller, the resolution requirement for the optical flow calculation result on the output side can be correspondingly reduced, but the fixed optical flow calculation method cannot adjust the output resolution of the optical flow. The same is true for the changes in the accuracy requirements of optical flow calculation caused by changes in other conditions such as frame rate. Therefore, the fixed optical flow calculation method in the related art cannot make corresponding adjustments in scenarios where the image resolution, frame rate, etc. change to achieve the best effect of the algorithm.

[0035] To solve the above problems, the present application provides an optical flow calculation method. The method is applied in a terminal device. The terminal device includes an application processor for running an operating system. The operating system of the terminal device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, taking the Android system with a layered architecture as an example, the software structure of the terminal device is exemplarily described.

[0036] Figure 1 It is a software structure block diagram of a terminal device provided in an embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into six layers, from top to bottom are the application layer, the application framework layer, Android runtime and system libraries, the hardware abstraction layer, the kernel layer, and the hardware layer. The application layer may include a series of application packages.

[0037] As Figure 1 shown, the application packages may include applications such as a camera, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, a short message, etc.

[0038] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0039] As Figure 1As shown in the figure, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0040] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0041] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0042] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0043] The phone manager is used to provide the communication function of the terminal device. For example, the management of call states (including answering, hanging up, etc.).

[0044] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0045] The notification manager enables application programs to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scroll bar text, such as the notification of a background-running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompting text information in the status bar, emitting a prompt sound, vibrating the electronic device, flashing the indicator light, etc.

[0046] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0047] The core library contains two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android.

[0048] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0049] The system library may include multiple functional modules. For example: SurfaceFlinger, Surface Manager, Media Libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0050] SurfaceFlinger is a graphics drawing engine for drawing graphics.

[0051] The Surface Manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0052] The Media Libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The Media Libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0053] The 2D graphics engine is a graphics drawing engine for 2D drawing.

[0054] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0055] The Hardware Abstraction Layer is a layer between the system library and the kernel layer. The Hardware Abstraction Layer at least includes a Graphics Processing Unit (GPU) and a Hardware Composer (HWC) module.

[0056] The fingerprint module is used to report fingerprint events to the fingerprint service in the application framework layer.

[0057] The HWC module is a driver abstraction layer for a dedicated chip for layer composition. For example, in the embodiments of the present application, the HWC module is used to connect the SurfaceFlinger and the display driver, that is, the HWC module is a communication bridge between the SurfaceFlinger and the display driver, so that the layers drawn by the SurfaceFlinger can be transmitted to the display driver for display.

[0058] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, a touch driver, a fingerprint driver, an audio driver, and a sensor driver.

[0059] The hardware layer at least includes a Display Driver Integrated Circuit (DDIC), an ambient light sensor, a touch sensor, and a display screen. The display driver chip is used to send drive signals and data to the display panel, and by controlling the screen brightness and color, make image contents such as pictures and letters presented on the screen. The light sensor is used to sense the light intensity or color of the surrounding environment. The touch sensor is used to sense the touch operations of the user on the display screen.

[0060] The following will combine Figure 2 Specifically introduce the optical flow calculation method of the embodiments of this application. The optical flow calculation method can be executed in the hardware abstraction layer or the kernel layer of the terminal device. For example, the optical flow calculation method can be executed in the GPU of the hardware abstraction layer of the terminal device. Figure 2 The exemplary method includes one or more steps, but does not constitute a limitation to this application. In addition, the order of the steps of the method is only an example, and the order of the steps can be changed. Without departing from the content disclosed in this application, additional steps can be added or steps can be reduced. The method specifically includes the following steps.

[0061] Step S201, obtain two adjacent frames of images in the image sequence.

[0062] In an embodiment of this application, the GPU of the terminal device obtains a set of input frame sequence images, and obtains two adjacent frames of images from this set of frame sequence images. For example, the GPU obtains a set of frame sequence images captured by the camera, and obtains two adjacent frames of images from the frame sequence images, or the GPU obtains a set of frame sequence images from the video stored in the terminal device, and obtains two adjacent frames of images from the frame sequence images.

[0063] Step S202, perform texture mapping (Mipmap) on each frame of image to obtain an image pyramid for each frame of image, where the image pyramid includes multiple image layers.

[0064] In an embodiment of this application, performing texture mapping on each frame of image to obtain an image pyramid for each frame of image includes: sequentially scaling each frame of image according to a preset equal ratio to obtain multiple image layers, each image layer is an image obtained by scaling the frame image according to the equal ratio, and all the image layers of each frame of image constitute the image pyramid of each frame of image. In an embodiment of this application, the preset equal ratio is 2. For example, refer to Figure 3, the pixels of the original image of a frame are 256x256. The original image of a frame is successively scaled according to a preset equal-value ratio to obtain a set of sub-images with pixels of 128x128, 64x64, 32x32, 16x16, 8x8, 4x4, 2x2, and 1x1 respectively. Among them, the original image of a frame is used as the 0th image layer of the image pyramid, the sub-image with pixels of 128x128 obtained after the first scaling is used as the 1st image layer of the image pyramid, the sub-image with pixels of 64x64 obtained after the second scaling is used as the 2nd image layer of the image pyramid, the sub-image with pixels of 32x32 obtained after the third scaling is used as the 3rd image layer of the image pyramid, the sub-image with pixels of 16x16 obtained after the fourth scaling is used as the 4th image layer of the image pyramid, the sub-image with pixels of 8x8 obtained after the fifth scaling is used as the 5th image layer of the image pyramid, the sub-image with pixels of 4x4 obtained after the sixth scaling is used as the 6th image layer of the image pyramid, the sub-image with pixels of 2x2 obtained after the seventh scaling is used as the 7th image layer of the image pyramid, and the sub-image with pixels of 1x1 obtained after the eighth scaling is used as the 8th image layer of the image pyramid. In an embodiment of the present application, the fewer the pixels of the image layer, the higher its position in the image pyramid.

[0065] Step S203, obtain the image parameter information of each frame of image, and determine an image layer in the image pyramid as the initial calculation layer according to the image parameter information. The image parameter information at least includes the image resolution.

[0066] In an embodiment of the present application, the terminal device inputs the image pyramid of each frame of image and the image parameter information of each frame of image into the compute shader of the GPU. The compute shader obtains the image resolution of each frame of image and determines an image layer of the image pyramid as the initial calculation layer according to the image resolution.

[0067] In an embodiment of the present application, determining an image layer of the image pyramid as the initial calculation layer according to the image parameter information includes: based on a preset image layer relationship table, determining the image layer in the image pyramid corresponding to the image resolution as the initial calculation layer according to the image resolution of each frame of image. In an embodiment of the present application, the image layer relationship table includes the corresponding relationships between multiple image resolutions and multiple initial calculation layers, and each image resolution corresponds to an initial calculation layer. In an embodiment of the present application, the compute shader looks up the image layer relationship table according to the image resolution of each frame of image to determine the initial calculation layer corresponding to the image resolution. Refer to Figure 4As shown, it is a schematic diagram of an image layer relationship table provided by an embodiment of the present application. The initial calculation layer corresponding to the frame image with an image resolution of 256x256 is the 5th image layer. The initial calculation layer corresponding to the frame image with an image resolution of 128x128 is the 4th image layer. The initial calculation layer corresponding to the frame image with an image resolution of 64x64 is the 3rd image layer. In an embodiment of the present application, if the first image resolution in the image parameter information is higher than the second image resolution, the position of the initial calculation layer of the frame image corresponding to the first image resolution in the image pyramid is higher than the position of the initial calculation layer of the frame image corresponding to the second image resolution in the image pyramid. In an embodiment of the present application, the larger the resolution of each frame image, the more details the image contains. The computer shader uses the image layer with a higher resolution in the image pyramid as the initial calculation layer, which can ensure the workload and accuracy of calculating the optical flow using the initial calculation layer.

[0068] Step S204: Obtain the preset output optical flow resolution, and determine an image layer of the image pyramid as the termination calculation layer according to the preset output optical flow resolution. Among them, the output optical flow resolution can be determined according to the size of the output optical flow.

[0069] In an embodiment of the present application, the output optical flow resolution is a preset constant value and can be set according to the needs of the user. In an embodiment of the present application, the computer shader calculates the output optical flow resolution of each image layer of the image pyramid, and uses the image layer corresponding to the output optical flow resolution that is consistent with the preset output optical flow resolution as the termination calculation layer. In an embodiment of the present application, calculating the output optical flow resolution of each image layer of the image pyramid includes: obtaining the image resolution of each image layer, and taking the ratio of the image resolution of each image layer to the preset block as the output optical flow resolution of each image layer. In an embodiment of the present application, the preset block is a pixel domain with a preset size, and the preset block can be set as needed, for example, set as a pixel domain of 8x8.

[0070] Step S205: Determine multiple calculation layers from the image pyramid according to the initial calculation layer and the termination calculation layer.

[0071] In an embodiment of the present application, the computer shader uses all the image layers between the initial calculation layer and the termination calculation layer in the image pyramid as the calculation layers. For example, referring to Figure 3 , if the computer shader determines that the 4th image layer in the image pyramid is the initial calculation layer and the 1st image layer is the termination calculation layer, then the 4th image layer, the 3rd image layer, the 2nd image layer, and the 1st image layer in the image pyramid are determined as the calculation layers.

[0072] Step S206: Sequentially calculate the output optical flow values of multiple calculation layers through a preset optical flow algorithm until an optical flow calculation result is obtained.

[0073] Reference Figure 5 As shown, in an embodiment of the present application, it is a flowchart of a method for sequentially calculating the output optical flow values of multiple calculation layers through a preset optical flow algorithm. The method specifically includes the following steps.

[0074] Step S501: Based on the initial calculation layer of two frames of images, use a preset optical flow algorithm to calculate the output optical flow value of the initial calculation layer.

[0075] In an embodiment of the present application, the preset optical flow algorithm includes: based on the initial calculation layer of two frames of images, use a first preset optical flow algorithm to calculate a first optical flow value; based on the first optical flow value, use a second preset optical flow algorithm to obtain a second optical flow value of the initial calculation layer of the two frames of images through iterative calculation, and use the second optical flow value as the output optical flow value of the initial calculation layer of the two frames of images.

[0076] Step S502: Use the output optical flow value of the initial calculation layer as the input data of the next image layer of the initial calculation layer, and use a preset optical flow algorithm to calculate the output optical flow value of the next calculation layer until the output optical flow value of the termination calculation layer is obtained, and use the output optical flow value of the termination calculation layer as the optical flow calculation result.

[0077] The above technical solution can adaptively calculate the optical flow of the input frame image with different image resolutions according to the preset optical flow algorithm without switching the algorithm, ensuring that the optical flow algorithm will not generate a large load when calculating the optical flow of images with different resolutions.

[0078] In an embodiment of the present application, the first preset optical flow algorithm is a block matching optical flow algorithm, and the second preset optical flow algorithm is a Lucas-Kanada sparse optical flow algorithm. In an embodiment of the present application, the two frames of images include the current frame image and the reference frame image. Reference Figure 6As shown in the figure, it is a schematic diagram of calculating optical flow by fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm provided by an embodiment of the present application. In an embodiment of the present application, based on the initial calculation layer of two frames of images, the first optical flow value is calculated using the first preset optical flow algorithm, including: dividing the initial calculation layer of the reference frame image and the initial calculation layer of the current frame image into a plurality of non-overlapping pixel blocks according to the size of the preset block; for each pixel block of the initial calculation layer of the current frame image, according to the preset matching criterion, performing a matching operation on all pixel blocks within the search range of the initial calculation layer of the reference frame image and each pixel block of the initial calculation layer of the current frame image, determining the pixel block with the smallest matching error with each pixel block of the initial calculation layer of the current frame image as the matching pixel block, calculating the optimal motion vector estimation value according to the relative position between each pixel block of the initial calculation layer of the current frame image and the corresponding matching pixel block, and determining the first optical flow value according to all the optimal motion vector estimation values. In an embodiment of the present application, the size of the preset block can be set to a pixel size of 8x8.

[0079] In an embodiment of the present application, the search range can be determined according to the image frame rate of the reference frame image. In an embodiment of the present application, the search range relationship table is searched according to the search range to determine the search range corresponding to the image frame rate of the frame image. The search range relationship table includes the relationships between a plurality of search ranges and a plurality of image frame rates, and each image frame rate corresponds to a search range.

[0080] In an embodiment of the present application, if the image frame rate of the reference frame image is greater than the preset image frame rate or greater than the image frame rate of the adjacent previous frame image of the reference frame image, the search range of the pixel blocks in the block matching optical flow algorithm is reduced according to the image frame rate of the reference frame image, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is reduced; if the image frame rate of the reference frame image is less than the preset image frame rate or less than the image frame rate of the previous frame image of the reference frame image, the search range of the pixel blocks in the matching optical flow algorithm is increased according to the image frame rate of the reference frame image, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is increased. For example, the image frame rate of the reference frame image is 30fps. If the image frame rate of the adjacent previous frame image of the reference frame image is 25fps, the pixel block of the matching optical flow algorithm corresponding to the previous frame image of the reference frame image is an 8*8 pixel domain, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is 8 times. Then, according to the image frame rate of the reference frame image, the search range of the pixel blocks in the matching optical flow algorithm is reduced from an 8*8 pixel domain to a 4*4 pixel domain, and the number of iterations of the Lucas-Kanada sparse optical flow algorithm is reduced from 8 times to 6 times. In an embodiment of the present application, the size of the search range of the matching optical flow algorithm and the number of iterations of the Lucas-Kanada sparse optical flow algorithm can be determined by looking up the corresponding parameter relationship table according to the image frame rate of the reference frame image. The parameter relationship table includes multiple different image frame rates, the sizes of the search ranges of multiple matching optical flow algorithms, and the corresponding relationships of the number of iterations of multiple Lucas-Kanada sparse optical flow algorithms.

[0081] Reference Figure 7 As shown, it is a schematic diagram of calculating the first optical flow value between the initial calculation layers of two frame images in an embodiment of the present application. As Figure 7As shown, image F1 is the initial calculation layer of the current frame image, and image F2 is the initial calculation layer of the reference frame image. Image F1 is divided into non-overlapping pixel blocks of size M*N pixels according to the size of the preset block. After a given search range (um, vm), matching operations are performed on all pixel blocks in image F1 according to the preset matching criterion. For example, for a pixel block with the center point coordinates (x, y) in image F1, the search range of the corresponding pixel block in image F2 is the area from (x - um, y - vm) to (x + um, y + vm). All pixel blocks within the search range of image F2 are matched with the pixel block in image F1 to obtain the pixel block with the minimum matching error. The difference between the starting point coordinates of the pixel block with the minimum matching error and the starting point coordinates (x, y) of the pixel block in image F1 is used as the optimal motion vector estimation value of the pixel block in image F1, where um and vm represent the number of pixels, and (x, y) represents pixel coordinates. In an embodiment of the present application, the preset matching criterion is the Sum of Absolute Differences algorithm.

[0082] In an embodiment of the present application, the second optical flow value of the initial calculation layers of two frames of images is obtained by iterative calculation using the second preset optical flow algorithm, including: based on a preset pixel window, performing multiple iterative calculations on all pixels of the initial calculation layers of the two frames of images, and adding the results of each iteration to the first optical flow value to obtain the second optical flow value. In an embodiment of the present application, the Lucas-Kanada sparse optical flow algorithm obtains the constraint equation of the image Ix*u + Iy*v + It = 0 based on the assumptions of brightness constancy (i.e., assuming that the brightness or gray value of a pixel is constant during movement), temporal persistence (i.e., assuming that the movement of an object on the image changes slowly over time), and spatial consistency (i.e., assuming that adjacent pixel points have similar movements), where Ix, Iy, and It represent the partial derivatives of the gray level of the pixel points in the image layer along the x, y, and t directions, (x, y) represents pixel coordinates, t represents time, and (u, v) represents the optical flow vector; based on the pixel window, performing multiple iterative calculations on the constraint equation Ix*u + Iy*v + It = 0 to obtain multiple iterative results of the optical flow, adding the results of each iteration of the optical flow to the first optical flow value to obtain the second optical flow value, or adding the last iterative result of the optical flow to the first optical flow value to obtain the second optical flow value.

[0083] In an embodiment of the present application, the computer shader uses the output optical flow value of the initial calculation layer as the input data of the next image layer of the initial calculation layer, and uses a preset optical flow algorithm to calculate the output optical flow value of the next calculation layer until the output optical flow value of the termination calculation layer is obtained, including: using the output optical flow value of the initial calculation layer as the initial optical flow value of the block matching optical flow algorithm, calculating the first optical flow value of the next calculation layer of two frames of images through the block matching optical flow algorithm, using the first optical flow value of the next calculation layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm to update the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, iteratively calculating the second optical flow value of the next calculation layer of two frames of images through the updated Lucas-Kanada sparse optical flow algorithm, using the second optical flow value of the next calculation layer as the output optical flow value of the next calculation layer, and sequentially performing iterative calculations layer by layer on multiple calculation layers until the output optical flow value of the termination calculation layer is calculated.

[0084] For example, the third image layer of the image pyramid is the initial calculation layer, and the first image layer of the image pyramid is the termination calculation layer. After calculating the output optical flow value of the initial calculation layer (i.e., the third image layer), use the output optical flow value of the initial calculation layer as the initial optical flow value of the block matching optical flow algorithm, calculate the first optical flow value of the second image layer through the block matching optical flow algorithm, use the first optical flow value of the second image layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm to update the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, iteratively calculate the second optical flow value of the second image layer of two frames of images through the Lucas-Kanada sparse optical flow algorithm, and use the second optical flow value of the second image layer as the initial optical flow value of the block matching optical flow algorithm to update the initial optical flow value of the block matching optical flow algorithm; based on the updated block matching optical flow algorithm, calculate the first optical flow value of the first image layer, use the first optical flow value of the first image layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm to update the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, and based on the updated Lucas-Kanada sparse optical flow algorithm, calculate the second optical flow value of the first image layer of two frames of images to obtain the output optical flow value of the first image layer, and use the output optical flow value of the first image layer as the output optical flow value of the termination calculation layer. Refer to Figure 5 , after calculating the output optical flow value of each image layer, output can be performed.

[0085] In an embodiment of the present application, the output optical flow value of the termination calculation layer can be used to identify actions in a frame image, or for video tracking and video frame interpolation. For example, in an embodiment of the present application, the frame image to be identified and the output optical flow value are respectively input into the ResNeXt3D subnet of the video action recognition model to determine the image features of the frame image and the optical flow features of the output optical flow value. The image features and the optical flow features are respectively input into the bidirectional attention layer of the video action recognition model to determine the first weighted feature after weighting the image features and the second weighted feature after weighting the optical flow features. The first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the user action corresponding to the user included in the frame image to be identified.

[0086] In the related art, when calculating optical flow using the block matching optical flow algorithm, through the hierarchical processing of the image pyramid, the problem of easy incorrect matching in large displacement and motion occlusion areas of the block matching optical flow algorithm can be optimized. Therefore, the block matching optical flow algorithm is applicable to larger motion scenarios, but the calculation accuracy of the block matching optical flow algorithm is relatively low (at the pixel level) and it is prone to falling into the local optimal matching problem. When calculating optical flow using the Lucas-Kanade sparse optical flow algorithm, the local information is derived from a small window (i.e., pixel window) of the point set around the image feature points for solution to obtain the optical flow result of the feature point set. Therefore, the Lucas-Kanade sparse optical flow algorithm is applicable to micro motion scenarios and has a relatively high calculation accuracy (at the sub-pixel level), but it is necessary to obtain image feature point information, increasing the calculation load of the algorithm. In the above embodiment of the present application, according to the image resolution, an image layer of the image pyramid is determined as the initial calculation layer, and according to the resolution of the output optical flow, the termination calculation layer of the image pyramid is determined. For the initial calculation layer of the image pyramid, first, the first optical flow value between two frame images is calculated through the block matching optical flow algorithm as a rough optical flow estimate, and then the optical flow estimate is used as the initial optical flow value of the Lucas-Kanade sparse optical flow algorithm, and the second optical flow value is calculated through the Lucas-Kanade sparse optical flow algorithm. By combining the first optical flow value and the first optical flow value, the output optical flow value of the initial calculation layer is obtained. The output optical flow value is used as the input data of the next image layer of the initial calculation layer, and the output optical flow value of each image layer is calculated layer by layer until the output optical flow value of the termination calculation layer is obtained as the optical flow calculation result. In this way, through the fusion processing of the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, the optical flow of the frame image is calculated. During the process of calculating the optical flow, there is no need to obtain image feature points for optical flow calculation, and at the same time, without switching the optical flow algorithm, the input frame images with different image resolutions can be adaptively calculated for optical flow, ensuring that the optical flow algorithm will not generate a large load.

[0087] In an embodiment of the present application, the size of the pixel block of the block matching optical flow algorithm is the same as the size of the pixel window of the Lucas-Kanada sparse optical flow algorithm. For example, the size of both the pixel block and the pixel window is an 8*8 pixel domain. In this way, steps S501 and S502 can be processed in parallel in the computer shader of the GPU in the form of workgroups or threads. The number of workgroups or threads can be the ratio of the resolution of the image layer to the size of the pixel block.

[0088] In an embodiment of the present application, the size and search path of the search block (i.e., pixel block) of the block matching optical flow algorithm affect the performance of the algorithm. The number of iterations and other termination conditions of the Lucas-Kanada sparse optical flow algorithm affect the performance of the algorithm. In this embodiment, the computer shader is used for optical flow calculation after fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm. Fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm includes: dynamically adjusting the parameters of the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm, as well as their participation degrees according to the image resolution, the frame rate of the image, the matching error, or the output optical flow resolution to obtain better comprehensive performance, so as to further implement an adaptive algorithm in different scenarios to calculate the optical flow.

[0089] In an embodiment of the present application, the computer shader determines an image layer of the image pyramid as the initial calculation layer according to the image resolution. The larger the resolution of the input frame image, the more details the frame image contains. If the image resolution is high, the computer shader adjusts the image layer with a lower position in the image pyramid to the initial calculation layer according to the image resolution. If the image resolution is low, the computer shader also adjusts the image layer with a lower position in the image pyramid to the initial calculation layer, so as to ensure that the workload and accuracy are certain when fusing the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow of the initial calculation layer.

[0090] In an embodiment of the present application, the computer shader performs fusion processing on each image layer from the initial calculation layer to the termination calculation layer through the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to obtain the output optical flow value of the corresponding image layer. If there are more image layers between the initial calculation layer and the termination calculation layer in the image pyramid, the load of the algorithm is greater. To balance the load of the algorithm, when the number of image layers between the initial calculation layer and the termination calculation layer exceeds the preset number, the computer shader uses the (N-1)th image layer below the initial calculation layer in the image pyramid as the target image layer, where N is the preset number (for example, 4); it processes each image layer from the initial calculation layer to the target image layer through the block matching optical flow algorithm and the Lucas-Kanada sparse optical flow algorithm to obtain the output optical flow value of the corresponding image layer, and layer-by-layer transfers the output optical flow value of the target image layer to the next image layer of the target image layer until it reaches the termination image layer. During the process of transferring the output optical flow value of the target image layer, there is no need to calculate the output optical flow value of the next transferred image layer, and only adjustment according to the preset multiple is required. For example, in an embodiment, when the output optical flow value of the target image layer is transferred in each image layer, the output optical flow value of the target image layer is multiplied by the preset multiple, where the preset multiple is the size ratio of adjacent two image layers. For example, the preset multiple can be set to 2, which is only for illustrative purposes and is not limited to this in actual applications.

[0091] In an embodiment of the present application, the smaller the frame interval of the input frame image, the higher the image frame rate of the input frame image and the smaller the motion amount of two frames of images. In an embodiment of the present application, if the frame interval of the two input images becomes smaller / the image frame rate becomes larger, it means that the requirement for the calculation accuracy of the optical flow becomes lower, and the computer shader reduces the search range of pixel blocks in the block matching optical flow algorithm and reduces the number of iterations of the Lucas-Kanada sparse optical flow algorithm, thereby accelerating the calculation and reducing the overall load of the algorithm. If the frame interval becomes larger / the image frame rate becomes lower, the requirement for the calculation accuracy of the optical flow becomes higher, and the computer shader increases the search range of pixel blocks in the matching optical flow algorithm and increases the number of iterations of the Lucas-Kanada sparse optical flow algorithm to meet the requirement for the calculation accuracy of the optical flow.

[0092] In an embodiment of the present application, during the process of calculating the output optical flow of each image layer, the computer shader determines whether to execute the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow according to the magnitude of the minimum matching error searched by the block matching optical flow algorithm. In an embodiment of the present application, determining whether to execute the Lucas-Kanada sparse optical flow algorithm to calculate the optical flow according to the magnitude of the minimum matching error searched by the block matching optical flow algorithm includes: if the minimum matching error is within the first preset error range, the Lucas-Kanada sparse optical flow algorithm is not executed to calculate the optical flow; if the minimum matching error is within the second preset error range, when executing the Lucas-Kanada sparse optical flow algorithm, the number of iterations of the Lucas-Kanada sparse optical flow algorithm for calculating the optical flow is reduced, or a preset error threshold is used as the early termination condition for the iteration of the Lucas-Kanada sparse optical flow algorithm. The first preset error range and the second preset error range can be set according to user needs, where the first preset error range is smaller than the second preset error range.

[0093] In an embodiment of the present application, when the computer shader determines that the current image layer of the image pyramid has received the output optical flow value transmitted by the previous image layer of the image pyramid, it first filters the received output optical flow value, and then determines the initial matching error of the current image layer according to the filtered output optical flow value. The search path and search range of the block matching optical flow method are adjusted according to the initial matching error, so that the initial matching error is reduced, and the matching operation of the block matching optical flow algorithm is quickly completed, improving the calculation efficiency.

[0094] The terminal device 100 involved in the embodiments of the present application will be introduced below.

[0095] Reference Figure 8 As shown, it is a schematic diagram of the hardware structure of the terminal device 100 provided by an embodiment of the present application. The terminal device 100 may include a processor 110, an internal memory 130, an external memory interface 120, a display screen 140, and a sensor module 150. The sensor module 150 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0096] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0097] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0098] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0099] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0100] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0101] It should be understood that the interface connection relationships among the modules illustrated in the embodiments of the present invention are only illustrative descriptions and do not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection manners in the above embodiments, or a combination of multiple interface connection manners.

[0102] The terminal device 100 realizes the display function through the GPU, the display screen 140, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 140 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0103] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel may adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 140, where N is a positive integer greater than 1.

[0104] The internal memory 130 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally referred to as DDR5 SDRAM), etc. The non-volatile memory may include disk storage devices, flash memory. Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, and can be divided into single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the number of potential levels of storage units. According to the storage specification, it can include universal flash storage (UFS), embedded multi media Card (eMMC), etc.

[0105] The random access memory can be directly read and written by the processor 110, can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data, etc.

[0106] The non-volatile memory can also store executable programs and store user and application data, etc., and can be pre-loaded into the random access memory for the processor 110 to directly read and write.

[0107] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0108] The internal memory 130 or the external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include a plurality of instructions. When the plurality of instructions are executed by the processor 110, the optical flow calculation method executed on the terminal device 100 in the above embodiments can be implemented to realize the optical flow calculation function of the terminal device 100.

[0109] This embodiment also provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the above related steps to implement the optical flow calculation method in the above embodiments.

[0110] In addition, some embodiments of the present application also provide a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. Among them, the memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory so that the chip executes the optical flow calculation method in each of the above method embodiments.

[0111] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0113] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0114] The unit described as a separate component may or may not be physically separated. The component shown as a unit may be a physical unit or multiple physical units, that is, it may be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in each embodiment of the present application, each functional unit may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0115] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solutions of some embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0116] The above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the above preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An optical flow calculation method, characterized in that, The method includes: Obtaining two adjacent frames of images in an image sequence; Performing texture mapping on each frame of image to obtain an image pyramid of each frame of image, where the image pyramid includes multiple image layers; Obtaining image parameter information of each frame of image, and determining an image layer of the image pyramid as an initial calculation layer according to the image parameter information, where the image parameters include image resolution; Obtaining a preset output optical flow resolution, and determining an image layer of the image pyramid as a termination calculation layer according to the preset output optical flow resolution; Determining multiple calculation layers from the image pyramid according to the initial calculation layer and the termination calculation layer; Sequentially calculating output optical flow values of the multiple calculation layers through a preset optical flow algorithm until an optical flow calculation result is obtained, including: Based on the initial calculation layer of the two frames of images, calculating an output optical flow value of the initial calculation layer of the two frames of images by using the preset optical flow algorithm; Taking the output optical flow value of the initial calculation layer of the two frames of images as input data of the next calculation layer of the initial calculation layer of the two frames of images, and calculating an output optical flow value of the next calculation layer by using the preset optical flow algorithm until an output optical flow value of the termination calculation layer is obtained, and taking the output optical flow value of the termination calculation layer as the optical flow calculation result.

2. The optical flow calculation method according to claim 1, wherein The preset optical flow algorithm includes: Based on the initial calculation layer of the two frames of images, calculating a first optical flow value by using a first preset optical flow algorithm; Based on the first optical flow value, obtaining a second optical flow value of the initial calculation layer of the two frames of images through iterative calculation by using a second preset optical flow algorithm, and taking the second optical flow value as the output optical flow value of the initial calculation layer of the two frames of images.

3. The optical flow calculation method according to claim 2, wherein The first preset optical flow algorithm is a block matching optical flow algorithm, and the second preset optical flow algorithm is a Lucas-Kanada sparse optical flow algorithm.

4. The optical flow calculation method according to claim 2 or 3, characterized in that, The two frames of images include a reference frame image and a current frame image. The calculating a first optical flow value based on the initial calculation layer of the two frames of images by using a first preset optical flow algorithm includes: Respectively dividing the initial calculation layer of the reference frame image and the initial calculation layer of the current frame image into a plurality of non-overlapping pixel blocks according to the size of a preset block; For each pixel block of the initial calculation layer of the current frame image, performing a matching operation on all pixel blocks within the search range of the initial calculation layer of the reference frame image and each pixel block of the initial calculation layer of the current frame image according to a preset matching criterion, and determining a pixel block with the smallest matching error with each pixel block of the initial calculation layer of the current frame image as a matching pixel block; Calculating an optimal motion vector estimation value according to the relative position between each pixel block of the initial calculation layer of the current frame image and the corresponding matching pixel block, and determining the first optical flow value according to all optimal motion vector estimation values.

5. The optical flow calculation method according to claim 4, wherein The method further includes: Determining the search range according to the image frame rate of the reference frame image.

6. The optical flow calculation method according to claim 5, characterized in that The determining the search range according to the image frame rate of the reference frame image includes: If the image frame rate of the reference frame image is greater than the preset image frame rate or greater than the image frame rate of the adjacent previous frame image of the reference frame image, reduce the search range of the pixel blocks in the block matching optical flow algorithm and reduce the number of iterations of the Lucas-Kanada sparse optical flow algorithm; If the image frame rate of the reference frame image is less than the preset image frame rate or less than the image frame rate of the adjacent previous frame image of the reference frame image, increase the search range of the pixel blocks in the matching optical flow algorithm and increase the number of iterations of the Lucas-Kanada sparse optical flow algorithm.

7. The optical flow calculation method according to claim 2, characterized in that The obtaining the second optical flow value of the initial calculation layer of the two frames of images by iteratively calculating using the second preset optical flow algorithm includes: Based on a preset pixel window, perform multiple iterative calculations on all pixels of the initial calculation layer of the two frames of images, and add the result of each iteration to the first optical flow value to obtain the second optical flow value.

8. The optical flow calculation method according to claim 2, wherein The using the output optical flow value of the initial calculation layer as the input data of the next image layer of the initial calculation layer and calculating the output optical flow value of the next calculation layer using the preset optical flow algorithm until the output optical flow value of the termination calculation layer is obtained includes: Use the output optical flow value of the initial calculation layer as the initial optical flow value of the block matching optical flow algorithm, and calculate the first optical flow value of the next calculation layer of the two frames of images through the block matching optical flow algorithm; Use the first optical flow value of the next calculation layer as the initial optical flow value of the Lucas-Kanada sparse optical flow algorithm, iteratively calculate the second optical flow value of the next calculation layer of the two frames of images through the updated Lucas-Kanada sparse optical flow algorithm, and use the second optical flow value of the next calculation layer as the output optical flow value of the next calculation layer, and perform layer-by-layer iterative calculations on the multiple calculation layers in sequence until the output optical flow value of the termination calculation layer is calculated.

9. The optical flow calculation method according to claim 1, wherein The determining an image layer of the image pyramid as the initial calculation layer according to the image parameter information includes: Based on a preset image layer relationship table, determine the corresponding image layer as the initial calculation layer according to the image resolution of each frame of image. The image layer relationship table includes the corresponding relationships between multiple image resolutions and multiple initial calculation layers, and each image resolution corresponds to an initial calculation layer.

10. The optical flow calculation method according to claim 1, characterized in that, The determining an image layer of the image pyramid as the termination calculation layer according to the preset output optical flow resolution includes: Calculate the output optical flow resolution of each image layer of the image pyramid, and use the image layer corresponding to the output optical flow resolution that is consistent with the preset output optical flow resolution as the termination calculation layer.

11. The optical flow calculation method according to claim 10, characterized in that The calculating the output optical flow resolution of each image layer of the image pyramid includes: Obtain the image resolution of each image layer, and use the ratio of the image resolution of each image layer to a preset block as the output optical flow resolution of each image layer. The preset block is a pixel domain with a preset size.

12. The optical flow calculation method according to claim 1, characterized in that The performing texture mapping on each frame of image to obtain the image pyramid of each frame of image includes: Each of the frames of images is successively scaled according to a preset equal-value ratio to obtain the multiple image layers, and each image layer is an image obtained by scaling each frame of image according to the equal-value ratio. All the image layers of each frame of image constitute the image pyramid of each frame of image.

13. The optical flow calculation method according to claim 1, wherein Determining an image layer of the image pyramid as the initial calculation layer according to the image parameter information includes: If the first image resolution in the image parameter information is higher than the second image resolution, the position of the frame image corresponding to the first image resolution in the initial calculation layer of the image pyramid is higher than the position of the initial calculation layer of the frame image corresponding to the second image resolution in the image pyramid.

14. A terminal device, characterized in that, The terminal device includes a memory and a processor: The memory is used for storing program instructions; The processor is used for reading and executing the program instructions stored in the memory. When the program instructions are executed by the processor, the terminal device executes the optical flow calculation method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions. When the program instructions run on the terminal device, the terminal device executes the optical flow calculation method according to any one of claims 1 to 13.

Images