Image motion estimation method and related device

Abstract

The invention provides an image motion estimation method and a related device, and an electronic device can intercept a rendering instruction for an image frame, obtains the related resources of the image frame based on the rendering instruction, carries out the motion estimation of two adjacent image frames through the fusion of an optical flow motion estimation method and a 3D motion field motion estimation method, and achieves the motion estimation of the two adjacent image frames. And determining motion vector information between two adjacent image frames. In this way, the motion estimation effect between two adjacent image frames can be improved.

Description

Image motion estimation method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 6, 2023, with application number 202311161813.4 and application name “A Method for Image Motion Estimation and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image motion estimation method and related devices. Background Art

[0003] The rapid development of mobile devices and the mobile internet has brought significant convenience to people's lives. Applications such as games and videos have become important ways for people to work, relax, and entertain themselves. To improve the smoothness of these applications, motion estimation is required on the original image frames to obtain motion information. Based on this motion information and the original image frames, new target image frames are generated. Therefore, the quality of the generated target image frames is closely related to the performance of the motion estimation of the original image frames. Currently, motion estimation algorithms perform poorly, resulting in low-quality target image frames.

[0004] Summary of the Invention

[0005] This application provides an image motion estimation method and related device that can integrate optical flow motion estimation and 3D motion field motion estimation to perform motion estimation on two adjacent image frames and determine the motion vector information between the two adjacent image frames. This can improve the accuracy of the motion estimation results between two adjacent image frames.

[0006] In a first aspect, the present application provides an image motion estimation method, which is applied to an electronic device, wherein the electronic device is installed with a first application, the method comprising: obtaining a rendering instruction issued by the first application, the rendering instruction being used to instruct a graphics processor (GPU) to render an image frame of the first application, the image frame of the first application comprising a first image frame and a second image frame; identifying from the rendering instruction the pixel position of a first object in the first image frame and the pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; identifying from the rendering instruction the pixel position of the first object in the second image frame and the pixel position of the second object in the second image frame; determining, by an optical flow motion estimation method, a motion vector of a pixel position of the first object between the first image frame and the second image frame; obtaining native motion vector information of the first application, and determining from the native motion vector information the motion vector of a pixel position of the second object between the first image frame and the second image frame.

[0007] The image motion estimation method provided by this application can intercept the rendering instructions of each rendering object in the image frame and use different motion estimation methods for different types of rendering objects. In this way, different motion estimation methods are applicable to different rendering objects. For example, the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. However, when the object in the picture frame moves quickly, the results of the optical flow motion estimation will have errors. The 3D motion field motion estimation method can also ensure relatively accurate motion estimation results when the object in the image frame moves quickly. However, the 3D motion field motion estimation method cannot perform motion estimation on semi-transparent objects. The native motion field vector information of the application is determined by the 3D motion field motion estimation method. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can improve the accuracy of the motion estimation results between two adjacent image frames.

[0008] In one possible implementation, the electronic device includes a motion estimation module; obtaining the rendering instruction issued by the first application specifically includes: the motion estimation module obtaining the rendering instruction issued by the first application from a graphics interface; obtaining the native motion vector information of the first application, and determining the motion vector of the pixel position of the second object between the first image frame and the second image frame from the native motion vector information, specifically includes: the motion estimation module obtaining the native motion vector information of the first application from the graphics interface, and determining the motion vector of the pixel position of the second object between the first image frame and the second image frame from the native motion vector information.

[0009] In this way, the native motion vector information of the application can be obtained directly from the graphics interface, so that the application does not need to be adapted and the application is unaware of the need to obtain the native motion vector information.

[0010] In one possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector information between the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; the motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

[0011] In this way, the optical flow algorithm interface on the system on chip (SoC) platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0012] In one possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame based on the pixel position of the first object in the first image frame, the pixel position of the second object in the second image frame, the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame to the motion estimation module; the motion estimation module uses the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame as the motion vector of the pixel position of the first object between the first image frame and the second image frame.

[0013] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0014] In one possible implementation, the method also includes: the motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; the motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; the optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

[0015] In a possible implementation, the native motion vector information is motion field motion vector information between the first image frame and the second image frame determined by the first application through a 3D motion field estimation method.

[0016] In one possible implementation, the electronic device also includes an interpolation module; the method also includes: the motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; the interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

[0017] In this way, the motion vector information between the first image frame and the second image frame can be used to perform frame interpolation, thereby saving power consumption of the electronic device.

[0018] In one possible implementation, before identifying the pixel position of the first object in the first image frame and the pixel position of the second object in the first image frame from the rendering instruction, the method also includes: the motion estimation module determines that the native motion vector information can be obtained from the graphics interface.

[0019] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and a neural network-based optical flow method.

[0020] In a possible implementation, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library for embedded systems OpenGL ES, and a drawing application programming interface Vulkan.

[0021] In a second aspect, the present application provides an image motion estimation method, which is applied to an electronic device, which has a first application installed; the method includes: obtaining a rendering instruction issued by the first application, the rendering instruction being used to instruct the GPU to render an image frame of the first application, the image frame of the first application including a first image frame and a second image frame; identifying the pixel position of a first object in the first image frame and the pixel position of a second object in the first image frame from the rendering instruction; identifying the pixel position of the first object in the second image frame and the pixel position of the second object in the second image frame from the rendering instruction; determining the motion vector of the pixel position of the first object between the first image frame and the second image frame through an optical flow motion estimation method; obtaining the first transformation matrix and first depth information of the first image frame, and the second transformation matrix and second depth information of the second image frame; based on the first transformation matrix, the first depth information, the second transformation matrix and the second depth information, determining the motion vector of the pixel position of the second object between the first image frame and the second image frame through a 3D motion field motion estimation method.

[0022] The image motion estimation method provided by the present application can intercept the rendering instructions of the game application for each rendering object in the image frame, and use different motion estimation methods for different types of rendering objects. In this way, different motion estimation methods are applicable to different rendering objects. For example, the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. The 3D motion field motion estimation method can also ensure relatively accurate motion estimation results when the object in the image frame moves at a fast speed. However, the motion vector of the dynamic object is relatively cumbersome to calculate using the 3D motion field estimation method, and the calculation time is relatively long. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can quickly determine the motion estimation results between two adjacent image frames.

[0023] In one possible implementation, the electronic device includes a motion estimation module; obtaining the rendering instruction issued by the first application specifically includes: the motion estimation module obtaining the rendering instruction issued by the first application from a graphics interface; obtaining the first transformation matrix and first depth information of the first image frame, and the second transformation matrix and second depth information of the second image frame specifically includes: the motion estimation module obtaining the first transformation matrix and first depth information of the first image frame, and the second transformation matrix and second depth information of the second image frame from the graphics interface.

[0024] In this way, the transformation matrix and depth information of the image frame in the application can be obtained directly from the graphics interface, so that the application does not need to adapt and the application is unaware of it, and the transformation matrix and depth information of the image frame can be obtained.

[0025] In one possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector information between the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; the motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

[0026] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0027] In one possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame based on the pixel position of the first object in the first image frame, the pixel position of the first object in the second image frame, the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame to the motion estimation module; the motion estimation module uses the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame as the motion vector of the pixel position of the first object between the first image frame and the second image frame.

[0028] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0029] In one possible implementation, the method also includes: the motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; the motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; the optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

[0030] In one possible implementation, the electronic device also includes an interpolation module; the method also includes: the motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; the interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

[0031] In this way, the motion vector information between the first image frame and the second image frame can be used to perform frame interpolation, thereby saving power consumption of the electronic device.

[0032] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and a neural network-based optical flow method.

[0033] In a possible implementation, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library for embedded systems OpenGL ES, and a drawing application programming interface Vulkan.

[0034] In a third aspect, the present application provides an image motion estimation method, which is applied to an electronic device, and the electronic device is installed with a first application; the method includes: obtaining a rendering instruction issued by the first application, the rendering instruction is used to instruct the GPU to render the image frame of the first application, and the image frame of the first application includes a first image frame and a second image frame; obtaining a first transformation matrix of the first image frame, a first depth information of the first image frame, a second transformation matrix of the second image frame, and a second depth information of the second image frame from the graphics interface; determining the optical flow motion vector information between the first image frame and the second image frame through an optical flow motion estimation method; the optical flow motion vector information includes an optical flow motion vector at a first pixel position in the second image; based on the first transformation matrix, the first depth information, the second transformation matrix, and the second depth information, determining the motion field motion vector information between the first image frame and the second image frame through a 3D motion field motion estimation method; the optical flow motion vector information includes the motion field motion vector at the first pixel position; if the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is less than a first specified threshold, taking the average of the optical flow motion vector at the first pixel position and the motion field vector at the first pixel position as the motion vector at the first pixel position.

[0035] The image motion estimation method provided in this application can perform motion estimation on two adjacent image frames by combining the transformation matrix and depth information of the two adjacent image frames with the optical flow motion estimation method and the 3D motion field motion estimation method, thereby determining the motion vector information between the two adjacent image frames. This can improve the accuracy of the motion estimation results.

[0036] In one possible implementation, if the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is greater than or equal to a first specified threshold, the depth value of the first pixel position and the depth values ​​of the surrounding pixel positions around the first pixel position in the second image frame are obtained from the second depth information; if the depth value at the first pixel position differs from the depth value of the surrounding pixel positions by less than a second specified threshold, the optical flow motion vector at the first pixel position is used as the motion vector at the first pixel position; if the depth value at the first pixel position differs from the depth value of the surrounding pixel positions by less than a second specified threshold, the motion field motion vector at the first pixel position is used as the motion vector at the first pixel position.

[0037] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and a neural network-based optical flow method.

[0038] In a possible implementation, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library for embedded systems OpenGL ES, and a drawing application programming interface Vulkan.

[0039] In a fourth aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, so that when the one or more processors execute the computer program, the method in any possible implementation of the above-mentioned first aspect is executed.

[0040] In a fifth aspect, the present application provides another electronic device, comprising one or more functional modules, which are used to execute the method in any possible implementation of the first aspect above.

[0041] In a sixth aspect, the present application provides a chip system comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method in any possible implementation of any of the above aspects.

[0042] In a seventh aspect, the present application provides a computer-readable storage medium comprising a computer program, which, when executed on a processor on an electronic device, enables execution of the method in any possible implementation of the first aspect described above.

[0043] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0045] FIG2 is a schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application;

[0046] FIG3 is a schematic diagram of a software system architecture interaction flow of an electronic device provided in an embodiment of the present application;

[0047] FIG4 is a schematic diagram of a flow chart of an image motion estimation method provided in an embodiment of the present application;

[0048] FIG5 is a schematic diagram of a template buffer of a first image frame provided in an embodiment of the present application;

[0049] FIG6 is a schematic diagram of a template buffer of a second image frame provided in an embodiment of the present application;

[0050] FIG7 is a schematic diagram of motion vector information between a first image frame and a second image frame provided by an embodiment of the present application;

[0051] FIG8A is a schematic diagram of generating a target image frame according to an embodiment of the present application;

[0052] FIG8B is a schematic diagram of generating another target image frame provided by an embodiment of the present application;

[0053] FIG9 is a schematic flow chart of an image motion estimation method provided in another embodiment of the present application;

[0054] FIG10 is a flowchart of an image motion estimation method provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0055] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0056] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0057] The following describes the hardware structure of an electronic device provided in an embodiment of the present application.

[0058] FIG1 shows a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application.

[0059] It should be understood that the electronic device 100 shown in FIG1 is merely an example, and the electronic device 100 may have more or fewer components than shown in FIG1, may combine two or more components, or may have a different component configuration. The various components shown in FIG1 may be implemented in hardware, including one or more signal processing and / or application specific integrated circuits, software, or a combination of hardware and software.

[0060] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K.

[0061] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0065] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0066] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0067] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 1.

[0068] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium- or high-frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs sound signals through an audio device (including but not limited to the speaker 170A, the receiver 170B, etc.) or displays images or videos through the display screen 194.

[0069] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0070] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0071] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0072] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0073] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0074] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0075] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0076] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0077] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs.

[0078] NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0080] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0081] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0082] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as a "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as a "handset", is used to convert audio electrical signals into sound signals. The microphone 170C, also known as a "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone jack 170D is used to connect wired headphones. The gyroscope sensor 180B can be used to determine the movement posture of the electronic device 100. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The touch sensor 180K, also known as a "touch panel", can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations applied on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to the user settings and function control of the electronic device 100. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195.

[0083] In the embodiment of the present application, the device type of the electronic device 100 can be any of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

[0084] The following uses a game application as an example to introduce some concepts related to motion estimation in the embodiments of the present application.

[0085] 1. Game interpolation.

[0086] During the running of a game application, electronic device 100 can perform motion estimation on multiple adjacent rendered frames to obtain motion vector information between them in order to reduce the frame rate of the game application's rendered image frames while ensuring the smoothness of the game screen. Based on the motion vector information and the multiple adjacent image frames, electronic device 100 can generate a target image frame. Electronic device 100 can insert the target image frame into a display queue for display.

[0087] 2. Optical flow motion estimation method.

[0088] Optical flow refers to the instantaneous speed of pixel motion of a moving object on the observation imaging plane. By using the changes in pixels between two adjacent image frames, the correspondence between the pixels in the latter image frame and the pixels in the previous image frame can be determined, thereby obtaining the optical flow information between the two adjacent image frames. This optical flow information between two adjacent image frames can be used to represent the motion vector information of the object between the two adjacent image frames.

[0089] Optical flow motion estimation is relatively robust and can be effective in most motion scenes. However, when the objects in the frame move quickly, the results of optical flow motion estimation will be inaccurate.

[0090] 3. 3D motion field motion estimation method.

[0091] The 3D motion field estimation method first determines the motion information of the same object in two adjacent image frames in 3D space. Then, the object's motion information in 3D space is projected onto a 2D plane to determine the motion vector information of the object between the two adjacent image frames.

[0092] The 3D motion field method can ensure relatively accurate motion estimation even when objects in an image frame are moving rapidly. However, it requires a large amount of computation to estimate the motion vectors of dynamic objects and cannot estimate the motion of semi-transparent objects.

[0093] Therefore, an embodiment of the present application provides an image motion estimation method. The electronic device 100 can intercept an application rendering instruction for an image frame and, based on the rendering instruction, obtain relevant resources of the image frame. The method integrates the optical flow motion estimation method and the 3D motion field motion estimation method to perform motion estimation on two adjacent image frames and determine the motion vector information between the two adjacent image frames. The electronic device 100 can generate a target image frame based on the motion vector information between the two adjacent image frames and the two adjacent image frames. This improves the accuracy of the motion estimation results between the two adjacent image frames.

[0094] The following describes a software system architecture of an electronic device provided in an embodiment of the present application.

[0095] FIG2 shows a schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application.

[0096] As shown in FIG2 , the software system architecture of the electronic device 100 may include an application (APP) layer, a native service (native) layer, and a kernel (kernel) layer.

[0097] The application layer includes one or more applications, such as gaming applications, extended reality (XR) applications, and autonomous driving applications. The application can be used to provide image content of an image frame and call a graphics interface to configure the GPU to render the image frame based on the image content of the image frame.

[0098] The native service layer may include a layer synthesizer (SurfaceFlinger), a motion estimation module, and one or more graphics interfaces, etc. Optionally, the native service layer may also include a frame insertion module and an optical flow algorithm interface. Among them, SurfaceFlinger is used to perform layer synthesis processing on the image frames rendered by the GPU, generate an interface including image frames, and send the interface including image frames to the display screen for display. The motion estimation module can be used to intercept the rendering instructions and frame sending instructions issued by the application, and determine the motion vector information between two adjacent image frames based on the relevant resources obtained by intercepting the rendering instructions and frame sending instructions.

[0099] The optical flow algorithm interface may be an algorithm interface provided by the system on chip (SoC) platform of the electronic device 100. The motion estimation module may call the optical flow algorithm interface to obtain optical flow motion vector information between two adjacent image frames. Specifically, when calling the optical flow algorithm interface, the motion estimation module may provide the two adjacent image frames to the optical flow algorithm interface. After obtaining the two adjacent image frames, the optical flow algorithm interface may calculate the optical flow motion vector information of the two adjacent image frames using an optical flow motion estimation method, and return the optical flow motion vector information of the two adjacent image frames to the motion estimation module.

[0100] The optical flow algorithm interface may be an OpenGL extension interface provided by the SOC platform. For example, the OpenGL extension interface may include a "glTexEstimateMotionQCOM" interface, a "glTexEstimateMotionRegionsQCOM" interface, etc.

[0101] In a possible implementation, the motion estimation module may also determine the optical flow motion vector information of two adjacent image frames by using an optical flow motion estimation method.

[0102] Among them, the optical flow motion estimation method may include: a pyramid optical flow method, a Lucas-Kanade method, an optical flow method based on a neural network (FlowNet / FlowNet2.0), and other optical flow algorithms.

[0103] Optionally, the motion estimation module may further send the motion vector between two adjacent image frames and the two adjacent image frames to the frame insertion module. The frame insertion module may generate a target image frame based on the motion vector between the two adjacent image frames and the two adjacent image frames.

[0104] The one or more graphics interfaces may include one or more of OpenGL (Open Graphics Library), OpenGL ES (Open Graphics Library for Embedded Systems), and Vulkan (Vulkan Application Programming Interface). The graphics interface may be used to configure a GPU to render image frames.

[0105] The kernel layer may include a graphics processing unit driver (GPU Driver), etc. The GPU Driver is used to drive the GPU to perform upper layer operations (eg, operations of rendering image frames configured by a graphics interface).

[0106] The following uses a game application as an example to describe the interaction process of each module in the above software system architecture.

[0107] FIG3 shows a schematic diagram of the software system architecture interaction flow of an electronic device provided in an embodiment of the present application.

[0108] As shown in Figure 3, the software system architecture interaction process is specifically described using a game application as an example. The game application may include a main thread (Main Thread) and a rendering thread (Render Thread).

[0109] The interaction process of each module can be as follows:

[0110] 1. The main thread of the game application can synchronize the image content to the rendering thread.

[0111] 2. After the rendering thread obtains the image content, it can send rendering instructions to the graphics interface.

[0112] The graphics interface may include any one of OpenGL, OpenGL ES, and Vulkan.

[0113] 3. The graphics interface can send rendering instructions to the motion estimation module.

[0114] 4. The motion estimation module can send rendering instructions to the GPU driver.

[0115] After receiving the rendering instruction, the GPU driver sends the rendering instruction to the GPU to configure the GPU to render an image frame based on the image content and store the rendered image frame in the buffer specified by the rendering instruction.

[0116] After rendering the image frame, the GPU can store the image frame in the buffer specified by the rendering instruction.

[0117] 5. After issuing the rendering instruction, the rendering thread can send the frame instruction to the graphics interface.

[0118] 6. The graphics interface can send frame instructions to the motion estimation module.

[0119] 7. The motion estimation module can forward the frame sending instruction to SurfaceFlinger.

[0120] After receiving the frame sending instruction, SurfaceFlinger can obtain the image frame buffer information from the frame sending instruction. Among them, when the GPU-rendered image frame is stored in the buffer, SurfaceFlinger can promptly know that the image frame has been stored in the buffer through the fence mechanism. Therefore, when the vertical synchronization (vsync) signal arrives, SurfaceFlinger can take the image frame from the image frame buffer for layer synthesis to synthesize the interface including the image frame, and then send the interface including the image frame to the display screen for display.

[0121] 8. The motion estimation module can determine the motion vector information between two adjacent image frames based on the relevant resources obtained by intercepting the rendering instruction and the frame sending instruction.

[0122] In one possible implementation, a motion estimation module may identify the type of one or more rendering objects in an image frame based on rendering instructions of the image frame. The motion estimation module may determine a motion estimation method corresponding to each of the one or more rendering objects based on the type of the one or more rendering objects. The motion estimation module may determine a motion vector for each of the one or more rendering objects based on the motion estimation method corresponding to the one or more rendering objects. The motion estimation module may determine motion vector information for the image frame based on the motion vectors of the one or more rendering objects and the pixel positions of the one or more rendering objects in the image frame. The motion vector information for the image frame includes a motion vector for each pixel position in the image frame.

[0123] In one possible implementation, the motion estimation module may retrieve two adjacent image frames from the image frame buffer indicated in the frame sending instruction based on the image frame rendering instruction, the transformation matrix and depth information of the two adjacent image frames, and the frame sending instruction. Based on the two adjacent image frames and the transformation matrix and depth information of the two adjacent image frames, the motion estimation module may combine optical flow motion estimation and 3D motion field motion estimation to determine motion vector information between the two adjacent image frames.

[0124] When the motion vector information between two adjacent image frames determined by the motion estimation module is used for frame interpolation, the following steps may be included:

[0125] 9. The motion estimation module provides the motion vector information between two adjacent image frames and the two adjacent image frames to the interpolation module.

[0126] 10. The interpolation module generates a target image frame based on the motion vector information between two adjacent image frames and the two adjacent image frames.

[0127] After acquiring the rendered image frame, the frame insertion module may determine a target image frame based on two adjacent image frames and motion vector information of the two adjacent image frames.

[0128] The frame insertion module may insert a target image frame between two adjacent image frames, or may generate a target image frame after two adjacent image frames.

[0129] For example, two adjacent image frames may include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame. When the interpolation module determines that a target image frame is to be inserted between the first image frame and the second image frame, the interpolation module may determine the target image frame based on the first image frame and the second image frame, and the motion vector information between the first image frame and the second image frame. The interpolation module may insert the target image frame between the first image frame and the second image frame and send it to SurfaceFlinger for synthesis and display.

[0130] When the interpolation module determines that a target image frame needs to be inserted after the second image frame, the interpolation module may determine the target image frame based on the second image frame and the motion vector information between the first image frame and the second image frame. The interpolation module may place the target image frame after the second image frame and send it to SurfaceFlinger for synthesis and display.

[0131] The interpolation module can store the target image frame into the buffer of the target image frame.

[0132] 11. The interpolation module can send the buffer address of the target image frame to SurfaceFlinger.

[0133] After obtaining the buffer of the target image frame, SurfaceFlinger can wait until the next vsync signal arrives, take out the target image frame from the buffer of the target image frame for layer synthesis, to synthesize the interface including the target image frame, and send the interface including the target image frame to the display screen for display.

[0134] The following describes motion estimation provided in the embodiments of the present application in conjunction with application scenarios.

[0135] Application scenario 1:

[0136] Because some game applications support functions that require motion estimation, such as motion blur, when the perspective of the image frame rotates in game scenarios, the game application will render objects such as characters or backgrounds in the image frame as if they were captured by a camera in high-speed motion, making the objects in the image frame appear blurry. To achieve this motion blur effect, when rendering the image frame, the game application can save the object's motion field motion vector by projecting the object's motion information in three-dimensional space onto a two-dimensional image. The game application can send the motion field motion vector information in the image frame to the graphics interface via rendering instructions. The motion field motion vector information in the image frame includes the motion field motion vector of the pixel position where opaque objects are located, but the motion field motion vector information in the image frame does not include the motion field motion vector of the pixel position where semi-transparent objects (e.g., shadows, skill special effects, etc.) are located. Therefore, the motion estimation module can intercept the rendering instructions of the game application for each rendered object in the image frame to obtain the motion field motion vector of the pixel position where each rendered object is located, and use the motion field motion vector of the pixel position where the opaque object is located as the motion vector of the pixel position where the opaque object is located. For semi-transparent objects, the motion estimation module can calculate the optical flow information between two adjacent image frames, and then determine the optical flow motion vector information between the two adjacent image frames from the optical flow information, and use the optical flow motion vector of the pixel position where the semi-transparent object is located as the motion vector of the pixel position where the opaque object is located.

[0137] FIG4 shows a flow chart of an image motion estimation method provided in an embodiment of the present application.

[0138] As shown in FIG4 , the image motion estimation method may include the following steps:

[0139] S401 : A motion estimation module may intercept, from a graphics interface, a rendering instruction of a game application for rendering an object in two adjacent image frames.

[0140] The two adjacent image frames include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame.

[0141] The game application can first send the rendering instruction of the first image frame to the graphics interface. After sending the rendering instruction of the first image frame, the game application can send the frame sending instruction of the first image frame to SurfaceFlinger through the graphics interface. After sending the frame sending instruction of the first image frame to SurfaceFlinger through the graphics interface, the game application can send the rendering instruction of the second image frame to the graphics interface. After sending the rendering instruction of the second image frame, the game application can send the frame sending instruction of the second image frame to SurfaceFlinger through the graphics interface.

[0142] The rendering instructions may include multiple instructions for configuring the graphics interface to call the GPU to render the image frame. The image frame is rendered in regions based on the objects within the image frame. Therefore, the rendering instructions may include a list of labels for multiple objects within the image frame and the pixel location of each object within the image frame.

[0143] S402: The motion estimation module may determine whether the rendering instruction contains a keyword of a semi-transparent object.

[0144] For example, the motion estimation module can intercept the "glLabelObjectEXT" instruction in the rendering instruction and determine whether the label carried by the "glLabelObjectEXT" instruction contains the keyword "translucent object". If so, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is a semi-transparent object. If not, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is an opaque object.

[0145] S403: If the rendering instruction includes a keyword of a semi-transparent object, the motion estimation module may determine that the rendering object is a semi-transparent object, and record pixel positions of the semi-transparent object in the first image frame and the second image frame respectively.

[0146] S404: If the rendering instruction does not include the keyword of the semi-transparent object, the motion estimation module may determine that the rendering object is an opaque object, and record the pixel positions of the opaque object in the first image frame and the second image frame respectively.

[0147] During the rendering of the first image frame, the motion estimation module can intercept the rendering instructions of the first image frame to identify and record the pixel positions of the semi-transparent objects in the first image frame. During the rendering of the second image frame, the motion estimation module can intercept the rendering instructions of the second image frame to identify and record the pixel positions of the semi-transparent objects in the second image frame.

[0148] Specifically, the motion estimation module can start the template test (stencil test) during the rendering of the first image frame and create a template buffer (stencil buffer) A. The [i, j]th template value in the template buffer A corresponds to the [i, j]th pixel position in the first image frame. Wherein, i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module can record the template value of the pixel position corresponding to the semi-transparent object in the first image frame in the template buffer A as St rans The template value of the pixel position corresponding to the opaque object in the first image frame is recorded as St opaq, for example, St rans Can take the value 0, St opaq The value can be 1.

[0149] For example, as shown in FIG5 , the pixel size of the first image frame may be 9*9, that is, there may be 9 rows*9 columns in the first image frame, for a total of 81 pixels. Therefore, the template buffer A may also include 9 rows*9 columns, for a total of 81 template values. Among them, objects 1 and 2 in the first image frame may be opaque objects, and objects at other pixel positions in the first image frame may be semi-transparent objects. Since the pixel positions of object 1 in the first image frame include the [2, 1]th pixel, the [2, 2]th pixel, the [3, 1]th pixel, and the [3, 2]th pixel. The pixel positions of object 2 in the first image frame include the [7, 1]th pixel, the [7, 2]th pixel, the [7, 1]th pixel, and the [7, 2]th pixel. Therefore, the template value at [2, 1]th, the template value at [2, 2]th, the template value at [3, 1]th, and the template value at [3, 2]th, as well as the template value at [7, 1]th, the template value at [7, 2]th, the template value at [7, 1]th, and the template value at [7, 2]th in the template buffer A can all be recorded as 1. Template values ​​at other positions in the template buffer A can be recorded as 0.

[0150] The motion estimation module can start the template test (stencil test) during the rendering of the second image frame and create a template buffer (stencil buffer B). The [i, j]th template value in the template buffer B corresponds to the [i, j]th pixel position in the second image frame, where i and j are positive integers. The motion estimation module can record the template value of the pixel position corresponding to the semi-transparent object in the second image frame in the template buffer B as St rans The template value of the pixel position corresponding to the opaque object in the second image frame is recorded as St opaq , for example, St rans The value of St can be 0. opaq The value of can be 1.

[0151] Exemplarily, as shown in FIG6 , the pixel size of the second image frame may be 9*9, that is, there may be 9 rows*9 columns in the second image frame, for a total of 81 pixels. Therefore, the template buffer B may also include 9 rows*9 columns, for a total of 81 template values. Among them, objects 1 and 2 in the second image frame may be opaque objects, and objects at other pixel positions in the second image frame may be semi-transparent objects. Since the pixel positions of object 1 in the second image frame include the [2, 3]th pixel, the [2, 4]th pixel, the [3, 3]th pixel, and the [3, 4]th pixel. The pixel positions of object 2 in the second image frame include the [5, 1]th pixel, the [5, 2]th pixel, the [6, 1]th pixel, and the [6, 2]th pixel. Therefore, the stencil value at positions [2, 3], [2, 4], [3, 3], and [3, 4] in the stencil buffer B, as well as the stencil value at positions [5, 1], [5, 2], [6, 1], and [6, 2] can all be recorded as 1. Stencil values ​​at other positions in the stencil buffer B can be recorded as 0.

[0152] S405 : The motion estimation module may determine the motion vector of the pixel position where the semi-transparent object is located between the first image frame and the second image frame by using an optical flow motion estimation method.

[0153] Among them, since the motion estimation module records the pixel positions of the semi-transparent objects in the first image frame and the second image frame respectively through the template buffer A and the template buffer B. Therefore, the motion estimation module can obtain the buffer of the first image frame and the buffer of the second image frame by intercepting the rendering instructions at the graphics interface. The motion estimation module can extract the rendered first image frame from the buffer of the first image frame and extract the rendered second image frame from the buffer of the second image frame. The motion estimation module can determine the pixel position of the semi-transparent object in the first image frame based on the template buffer A and the first image frame, and determine the pixel position of the semi-transparent object in the second image frame based on the template buffer B and the second image frame.

[0154] The motion estimation module can determine the optical flow information of the semi-transparent object between the first image frame and the second image frame through the optical flow motion estimation method based on the first image frame, the second image frame and the pixel positions of the semi-transparent object in the first image frame and the second image frame respectively.

[0155] The motion estimation module may determine a motion vector of a pixel position where the semi-transparent object is located between the first image frame and the second image frame based on optical flow information of the semi-transparent object between the first image frame and the second image frame.

[0156] Among them, the motion estimation module can determine the optical flow information of the semi-transparent object between the first image frame and the second image frame through optical flow motion estimation methods such as the pyramid optical flow method, the Lucas-Kanade method, and the neural network-based optical flow method (FlowNet / FlowNet2.0).

[0157] In one possible implementation, the motion estimation module can provide the first image frame and the second image frame to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface can calculate the optical flow motion vector information of the two adjacent image frames through the optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module. The motion estimation module can determine the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel positions of the semi-transparent object in the first image frame and the second image frame respectively.

[0158] In one possible implementation, the motion estimation module may provide the pixel positions of the first image frame, the second image frame, and the semi-transparent object in the first image frame and the second image frame, respectively, to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may determine the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame based on the pixel positions of the first image frame, the second image frame, and the semi-transparent object in the first image frame and the second image frame, respectively. The optical flow algorithm interface may return the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame to the motion estimation module.

[0159] S406 , the motion estimation module may obtain native motion vector information of the game application from the graphics interface, and determine the motion vector of the pixel position where the opaque object is located between the first image frame and the second image frame from the native motion vector information.

[0160] For example, the motion estimation module can obtain the native motion vector information of the game application from the graphics interface through the "glGetUniformLocation" instruction.

[0161] The motion estimation module can determine the motion vector of the pixel position of the opaque object between the first image frame and the second image frame from the native motion vector information based on the pixel position of the opaque object in the first image frame and the pixel position of the opaque object in the second image frame.

[0162] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S407 may be executed.

[0163] S407 : The frame insertion module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0164] Among them, after obtaining the motion vector information between the first image frame and the second image frame sent by the motion estimation module, the interpolation module can insert the target image frame between the first image frame and the second image frame, or insert the target image frame after the first image frame and the second image frame.

[0165] When the interpolation module determines to insert a target image frame between the first image frame and the second image frame, the interpolation module may generate the target image frame based on the first image frame and the second image frame, and motion vector information between the first image frame and the second image frame.

[0166] For example, as shown in FIG7 , the motion vector information between the first image frame and the second image frame may include the motion vector of object 1 and the motion vector of object 2. The pixel positions of object 1 in the first image frame include the [2, 1]th pixel, the [2, 2]th pixel, the [3, 1]th pixel, and the [3, 2]th pixel. The pixel positions of object 1 in the second image frame include the [2, 3]th pixel, the [2, 4]th pixel, the [3, 3]th pixel, and the [3, 4]th pixel. The motion vector of object 1 indicates that object 1 has moved 2 pixels to the right.

[0167] The pixel positions of object 2 in the first image frame include pixel [7, 1], pixel [7, 2], pixel [7, 1], and pixel [7, 2]. The pixel positions of object 2 in the second image frame include pixel [5, 1], pixel [5, 2], pixel [6, 1], and pixel [6, 2]. The motion vector of object 2 indicates that object 2 has moved upward by 2 pixels.

[0168] As shown in Figure 8A, when the interpolation module determines to insert a target image frame between the first image frame and the second image frame, from the first image frame to the target image frame, object 1 needs to be moved right by one pixel, and object 2 needs to be moved upward by one pixel. Therefore, the pixel positions of object 1 in the target image frame include the pixel [2, 2]th, the pixel [2, 3]th, the pixel [3, 2]th, and the pixel [3, 3]. The pixel positions of object 2 in the target image frame include the pixel [6, 1]th, the pixel [6, 2]th, the pixel [7, 1]th, and the pixel [7, 2].

[0169] When the frame interpolation module determines to generate the target image frame after the second image frame, the frame interpolation module may generate the target image frame based on the second image frame and motion vector information between the first image frame and the second image frame.

[0170] As shown in FIG8B , when the interpolation module determines that the target image frame should be generated after the second image frame, object 1 needs to be moved 2 pixels to the right, and object 2 needs to be moved 2 pixels upward from the second image frame to the target image frame. Therefore, the pixel positions of object 1 in the target image frame include the pixel [2, 5]th, the pixel [2, 6]th, the pixel [3, 5]th, and the pixel [3, 6]th. The pixel positions of object 2 in the target image frame include the pixel [3, 1]th, the pixel [3, 2]th, the pixel [4, 1]th, and the pixel [4, 2].

[0171] The above examples are only used to explain the present application and should not be construed as limiting.

[0172] An image motion estimation method is provided in an embodiment of the present application. By intercepting the rendering instructions of each rendering object in the image frame, different motion estimation methods can be used for different types of rendering objects. In this way, different motion estimation methods are applicable to different rendering objects. For example, the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. However, when the object in the picture frame moves quickly, the result of the optical flow motion estimation will have errors. The 3D motion field motion estimation method can also ensure relatively accurate motion estimation results when the object in the image frame moves quickly. However, the 3D motion field motion estimation method cannot perform motion estimation on translucent objects. The native motion field vector information of the application is determined by the 3D motion field motion estimation method. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can improve the accuracy of the motion estimation results between two adjacent image frames.

[0173] For example, an optical flow motion estimation method is used to determine the motion vector of a first object between two adjacent image frames, and the motion vector of the first object between the two adjacent image frames is obtained from the native motion vector of the application. In the embodiment of the present application, the first object can be a semi-transparent object, and the second object can be an opaque object. In other embodiments, other types of divisions can be used for the first and second objects.

[0174] Application Scenario 2:

[0175] Since the motion estimation module cannot directly obtain the motion vector of the moving field from the rendering instructions of the game application for each rendering object in the image frame, the motion estimation module can obtain the transformation matrix and depth information of two adjacent image frames by intercepting the rendering instructions of the game application for each rendering object in the image frame. The motion estimation module can determine the motion vector of the moving field at the pixel position of the static object between the two adjacent image frames through the transformation matrix and depth information of the two adjacent image frames. The motion estimation module can calculate the optical flow information between the two adjacent image frames and determine the motion vector of the pixel position of the dynamic object between the two adjacent image frames through the optical flow information.

[0176] FIG9 is a schematic flow chart showing a method for estimating image motion provided in another embodiment of the present application.

[0177] As shown in FIG9 , the image motion estimation method may include the following steps:

[0178] S901. A motion estimation module may intercept, from a graphics interface, rendering instructions for rendering objects in two adjacent image frames of a game application.

[0179] The two adjacent image frames include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame.

[0180] For the specific content, please refer to step S401 in the embodiment shown in FIG4 , which will not be described in detail here.

[0181] S902: The motion estimation module may determine whether the rendering instruction contains a keyword of a dynamic object.

[0182] For example, the motion estimation module can intercept the "glLabelObjectEXT" instruction in the rendering instruction and determine whether the label carried by the "glLabelObjectEXT" instruction contains a keyword for a dynamic object (e.g., a person, a monster, a shadow, etc.). If so, the motion estimation module can determine that the rendered object corresponding to the "glLabelObjectEXT" instruction is a dynamic object. If not, the motion estimation module can determine that the rendered object corresponding to the "glLabelObjectEXT" instruction is a static object.

[0183] S903: If the rendering instruction contains a keyword of a dynamic object, the motion estimation module may determine that the rendering object is a dynamic object, and record pixel positions of the dynamic object in the first image frame and the second image frame respectively.

[0184] S904: If the rendering instruction does not contain the keyword of the dynamic object, the motion estimation module may determine that the rendering object is a static object, and record the pixel positions of the static object in the first image frame and the second image frame respectively.

[0185] During the rendering of the first image frame, the motion estimation module can intercept the rendering instructions of the first image frame to identify and record the pixel positions of the dynamic objects in the first image frame. During the rendering of the second image frame, the motion estimation module can intercept the rendering instructions of the second image frame to identify and record the pixel positions of the dynamic objects in the second image frame.

[0186] Specifically, the motion estimation module can start the template test (stencil test) during the rendering of the first image frame and create a template buffer (stencil buffer) C. The [i, j]th template value in the template buffer C corresponds to the [i, j]th pixel position in the first image frame. Wherein, i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module can record the template value of the pixel position corresponding to the dynamic object in the first image frame in the template buffer C as St dyna The template value of the pixel position corresponding to the opaque object in the first image frame is recorded as St sta , for example, St dyna Can take the value 0, St sta The value can be 1.

[0187] In an example, the template value in the template buffer C may refer to the template value in the template buffer A in the embodiment shown in FIG. 5 .

[0188] The motion estimation module may start a stencil test during the rendering of the second image frame and create a stencil buffer D. The stencil value [i, j] in the stencil buffer D corresponds to the pixel position [i, j] in the second image frame. Where i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module may record the stencil value of the pixel position corresponding to the dynamic object in the second image frame in the stencil buffer D as St dyna The template value of the pixel position corresponding to the opaque object in the second image frame is recorded as St sta , for example, ST dyna The value can be 0, ST sta The value can be 1.

[0189] In an example, the template value in the template buffer D may refer to the template value in the template buffer B in the embodiment shown in FIG. 6 .

[0190] S905 : The motion estimation module may determine the motion vector of the pixel position where the dynamic object is located between the first image frame and the second image frame by using an optical flow motion estimation method.

[0191] Among them, since the motion estimation module records the pixel positions of the dynamic objects in the first image frame and the second image frame respectively through the template buffer C and the template buffer D, the motion estimation module can obtain the buffer of the first image frame and the buffer of the second image frame by intercepting the rendering instructions at the graphics interface. The motion estimation module can extract the rendered first image frame from the buffer of the first image frame and extract the rendered second image frame from the buffer of the second image frame. The motion estimation module can determine the pixel position of the dynamic object in the first image frame based on the template buffer C and the first image frame, and determine the pixel position of the dynamic object in the second image frame based on the template buffer D and the second image frame.

[0192] The motion estimation module can determine the optical flow information of the dynamic object between the first image frame and the second image frame through the optical flow motion estimation method based on the first image frame, the second image frame and the pixel positions of the dynamic object in the first image frame and the second image frame respectively.

[0193] The motion estimation module may determine a motion vector of a pixel position where the dynamic object is located between the first image frame and the second image frame based on optical flow information of the dynamic object between the first image frame and the second image frame.

[0194] Among them, the motion estimation module can determine the optical flow information of the dynamic object between the first image frame and the second image frame through optical flow motion estimation methods such as the pyramid optical flow method, the Lucas-Kanade method, and the neural network-based optical flow method (FlowNet / FlowNet2.0).

[0195] In one possible implementation, the motion estimation module can provide the first image frame and the second image frame to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface can calculate the optical flow motion vector information of the two adjacent image frames through the optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module. The motion estimation module can determine the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame based on the pixel position of the dynamic object in the first image frame and the second image frame, respectively, from the optical flow motion vector information between the first image frame and the second image frame.

[0196] In one possible implementation, the motion estimation module may provide the pixel positions of the first image frame, the second image frame, and the dynamic object in the first image frame and the second image frame, respectively, to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may determine the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame based on the pixel positions of the first image frame, the second image frame, and the dynamic object in the first image frame and the second image frame, respectively. The optical flow algorithm interface may return the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame to the motion estimation module.

[0197] S906 : The motion estimation module may obtain the transformation matrix and depth information of the first image frame, and the transformation matrix and depth information of the second image frame from the graphic interface.

[0198] Among them, the motion estimation module can obtain the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame by intercepting the "glBufferSubData" instruction or "glGetUniformLocation" instruction and other related instructions in the rendering instruction in the graphics interface.

[0199] The transformation matrix of the first image frame is used to represent the transformation relationship of the first image frame from the perspective coordinate system of the first image frame to the world coordinate system. The transformation matrix of the second image frame is used to represent the transformation relationship of the second image frame from the perspective coordinate system of the first image frame to the world coordinate system.

[0200] S907. The motion estimation module can determine the motion vector of the pixel position of the static object between the first image frame and the second image frame based on the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame through a 3D motion field motion estimation method.

[0201] Because the motion vector of a static object in the first image frame is generated by the perspective transformation between the first image frame and the second image frame, the motion estimation module can reproject the depth information of the second image frame onto the perspective coordinates of the first image frame to obtain the depth information of the second image frame in the perspective coordinate system of the first image frame. Based on the depth information of the second image frame in the perspective coordinate system of the first image frame and the depth information of the first image frame, the motion estimation module can determine the motion vector of the static object between the first and second image frames.

[0202] The calculation formula of the motion vector of the static object between the first image frame and the second image frame may be as follows:

[0203] In the above formula (1), v 3D-Sis the motion vector of the pixel position of the static object between the first image frame and the second image frame, VP n-1 is the transformation matrix of the first image frame, is the transposed matrix of the transformation matrix of the second image frame. cur-S is the depth value of the pixel position of the static object in the second image frame. pre-S is the depth value of the pixel position where the static object is located in the first image frame.

[0204] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S908 may be executed.

[0205] S908 : The frame insertion module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0206] For the specific content, please refer to step S407 in the embodiment shown in FIG4 , which will not be described in detail here.

[0207] The image motion estimation method provided by the present application can intercept the rendering instructions of the game application for each rendering object in the image frame, and use different motion estimation methods for different types of rendering objects. In this way, different motion estimation methods are applicable to different rendering objects. For example, the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. The 3D motion field motion estimation method can also ensure relatively accurate motion estimation results when the object in the image frame moves at a fast speed. However, the motion vector of the dynamic object is relatively cumbersome to calculate using the 3D motion field estimation method, and the calculation time is relatively long. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can quickly determine the motion estimation results between two adjacent image frames.

[0208] For example, an optical flow motion estimation method is used to determine the motion vector of the first object between two adjacent image frames, and a 3D motion field motion estimation method is used to determine the motion vector of the first object between two adjacent image frames. In an embodiment of the present application, the first object can be a dynamic object, and the second object can be a static object. In other embodiments, other types of divisions can be used for the first and second objects.

[0209] Application scenario 3:

[0210] Because the motion estimation module cannot directly obtain the motion vectors of the motion field from the rendering instructions of each rendered object in the image frame intercepted by the game application, and because some game applications define different labels for each rendered object in the image frame, the motion estimation module may not be able to identify the object type in the image frame. Therefore, the motion estimation module can use the transformation matrix and depth information of two adjacent image frames and integrate the optical flow motion estimation method and the 3D motion field motion estimation method to determine the motion vector information between two adjacent image frames.

[0211] FIG10 shows a flow chart of an image motion estimation method provided in another embodiment of the present application.

[0212] As shown in FIG10 , the image motion estimation method may include the following steps:

[0213] S1001. A motion estimation module may intercept a rendering instruction of two adjacent image frames from a game application through a graphics interface to obtain the two adjacent image frames and the transformation matrix and depth information of the two adjacent image frames.

[0214] Among them, the motion estimation module can obtain the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame by intercepting the "glBufferSubData" instruction or "glGetUniformLocation" instruction and other related instructions in the rendering instruction in the graphics interface.

[0215] The transformation matrix of the first image frame is used to represent the transformation relationship of the first image frame from the perspective coordinate system of the first image frame to the world coordinate system. The transformation matrix of the second image frame is used to represent the transformation relationship of the second image frame from the perspective coordinate system of the first image frame to the world coordinate system.

[0216] The depth information of the first image frame includes a depth value of each pixel position in the first image frame. The depth information of the second image frame includes a depth value of each pixel position in the second image frame.

[0217] S1002: The motion estimation module may determine optical flow motion vector information between the first image frame and the second image frame by using an optical flow motion estimation method.

[0218] The motion estimation module can determine the optical flow motion vector information between the first image frame and the second image frame through an optical flow motion estimation method such as a pyramid optical flow method, a Lucas-Kanade method, or a neural network-based optical flow method (FlowNet / FlowNet2.0).

[0219] In one possible implementation, the motion estimation module may provide the first image frame and the second image frame to an optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may calculate optical flow motion vector information of two adjacent image frames using an optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module.

[0220] The optical flow motion vector information between the first image frame and the second image frame may include the optical flow motion vector of each pixel position in the second image frame between the first image frame and the second image frame.

[0221] S1003. The motion estimation module can determine the motion field motion vector information between the first image frame and the second image frame through a 3D motion field motion estimation method based on the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame.

[0222] The motion estimation module may reproject the depth information of the second image frame onto the perspective coordinates of the first image frame to obtain the depth information of the second image frame in the perspective coordinate system of the first image frame. The motion estimation module may determine motion field motion vector information between the first and second image frames based on the depth information of the second image frame in the perspective coordinate system of the first image frame and the depth information of the first image frame.

[0223] The motion field motion vector information between the first image frame and the second image frame may include the motion field motion vector of each pixel position in the second image frame between the first image frame and the second image frame.

[0224] The calculation formula of the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame may be as follows:

[0225] In the above formula (2), v 3D is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, VP n-1 is the transformation matrix of the first image frame, is the transposed matrix of the transformation matrix of the second image frame. cur is the depth value of the first pixel position in the second image frame. pre The depth value of the first pixel position in the second image frame reprojected to the second pixel position in the first image frame.

[0226] The first pixel position may be any pixel position in the second image frame. Therefore, the motion vector of the motion field between the first image frame and the second image frame at any pixel position in the second image frame can be determined based on the above formula (2).

[0227] S1004: The motion estimation module may determine whether a difference between an optical flow motion vector at a first pixel position in the second image frame and a motion field motion vector at the first pixel position is less than a specified threshold 1.

[0228] If the difference between the optical flow motion vector at the first pixel position in the second image frame and the motion field motion vector at the first pixel position is less than the specified threshold 1, step S1005 is executed.

[0229] If the difference between the optical flow motion vector at the first pixel position in the second image frame and the motion field motion vector at the first pixel position is greater than or equal to the specified threshold 1, steps S1006 to S1009 are executed.

[0230] S1005: The motion estimation module may use an average value of the optical flow motion vector and the motion field motion vector at the first pixel position as the motion vector at the first pixel position.

[0231] If the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is less than a specified threshold of 1, it can be considered that the optical flow motion vector determined by the optical flow motion estimation method and the motion field motion vector determined by the 3D motion field motion estimation method at the first pixel position are consistent. Therefore, the average of the optical flow motion vector and the motion field motion vector at the first pixel position can be used as the motion vector at the first pixel position.

[0232] That is:

[0233] v out =(v 3D +v 2D ) / 2 formula (3)

[0234] In the above formula (3), v out is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, v 3D is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, v 2D is the optical flow motion vector of the first pixel position in the second image frame between the first image frame and the second image frame.

[0235] S1006: The motion estimation module may obtain a depth value of the first pixel position and depth values ​​of surrounding pixel positions around the first pixel position in the second image frame.

[0236] If the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is greater than or equal to a specified threshold of 1, it can be considered that the optical flow motion vector determined by the optical flow motion estimation method and the motion field motion vector determined by the 3D motion field motion estimation method at the first pixel position differ significantly. Therefore, it can be determined whether the difference between the depth value at the first pixel position and the depth values ​​of the surrounding pixel positions is less than a specified threshold of 2, thereby determining whether the motion vector at the first pixel position is the optical flow motion vector or the motion field motion vector.

[0237] S1007: The motion estimation module may determine whether the difference between the depth value of the first pixel position and the depth values ​​of surrounding pixel positions is less than a specified threshold 2.

[0238] If the depth value of the first pixel position differs from the depth values ​​of the surrounding pixel positions by less than the specified threshold 2, step S1008 is executed.

[0239] If the depth value of the first pixel position differs from the depth values ​​of the surrounding pixel positions by more than or equal to the specified threshold 2, step S1009 is executed.

[0240] The first pixel position has multiple surrounding pixel positions. For example, there may be 8 surrounding pixel positions around the first pixel position.

[0241] In one possible implementation, when determining whether the difference between the depth value at the first pixel position and the depth values ​​of the surrounding pixels is less than a specified threshold of 2, the depth value differences between the first pixel position and the plurality of surrounding pixel positions may be calculated. If the average of the depth value differences between the first pixel position and the plurality of surrounding pixel positions is less than the specified threshold of 2, it may be determined that the depth value at the first pixel position differs from the depth values ​​of the surrounding pixels by less than the specified threshold of 2.

[0242] In one possible implementation, when determining whether the difference between the depth value at the first pixel position and the depth values ​​of the surrounding pixels is less than a specified threshold of 2, the depth value differences between the first pixel position and the plurality of surrounding pixel positions may be calculated. If the depth value differences between the first pixel position and the plurality of surrounding pixel positions are all less than the specified threshold of 2, then it may be determined that the depth value at the first pixel position differs from the depth values ​​of the surrounding pixels by less than the specified threshold of 2.

[0243] S1008. The motion estimation module may use the optical flow motion vector at the first pixel position as the motion vector at the first pixel position.

[0244] S1009: The motion estimation module may use the motion field motion vector at the first pixel position as the motion vector at the first pixel position.

[0245] Since the depth values ​​of each pixel position in a semi-transparent object are not much different, if the depth value difference between the first pixel position and the surrounding pixel positions is small, it means that the area where the first pixel position is located is likely to be the background (that is, the semi-transparent object). For semi-transparent objects, the optical flow motion estimation method can be used to obtain an accurate motion vector. Therefore, if the depth value of the first pixel position differs from the depth value of the surrounding pixels by less than a specified threshold of 2, the optical flow motion vector v of the first pixel position can be used as the background. 2D , as the motion vector v at the first pixel position out , that is, v out =v 2D .

[0246] Since the depth values ​​of each pixel position in an opaque object are quite different, if the depth value of the first pixel position is significantly different from that of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be an opaque object. For opaque objects, the 3D motion field motion estimation method can be used to obtain an accurate motion vector. Therefore, if the depth value of the first pixel position differs from the depth value of the surrounding pixels by more than a specified threshold of 2, the motion field motion vector v of the first pixel position can be used as the motion vector v of the first pixel position. 3D . As the motion vector v of the first pixel position out That is, v out =v 2D .

[0247] In some embodiments, since the optical flow motion vectors of each pixel position in a semi-transparent object are not much different, if the optical flow motion vectors of the first pixel position are slightly different from those of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be the background (i.e., the semi-transparent object). For semi-transparent objects, an accurate motion vector can be obtained by using the optical flow motion estimation method. Therefore, if the difference between the optical flow motion vector of the first pixel position and the optical flow motion vectors of the surrounding pixels is less than a specified threshold of 3, the optical flow motion vector v of the first pixel position can be used as the background. 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0248] Since the optical flow motion vectors of each pixel position in an opaque object are quite different, if the optical flow motion vectors of the first pixel position are significantly different from those of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be an opaque object. For opaque objects, the 3D motion field motion estimation method can be used to obtain an accurate motion vector. Therefore, if the difference between the optical flow motion vector of the first pixel position and the optical flow motion vectors of the surrounding pixels is greater than or equal to the specified threshold value 3, the motion field motion vector v of the first pixel position can be used as the target. 3D , as the motion vector v at the first pixel positionout That is, v out =v 2D .

[0249] In other embodiments, the depth value difference and the optical flow motion vector difference of the first pixel position can be comprehensively considered to determine whether the first pixel position belongs to a semi-transparent object or an opaque object, thereby determining whether the motion vector of the first pixel position is an optical flow motion vector or a motion field motion vector.

[0250] If the depth value of the first pixel position differs from the depth value of the surrounding pixels by less than a specified threshold of 2, and the optical flow motion vector of the first pixel position differs from the optical flow motion vector of the surrounding pixels by less than a specified threshold of 3, the optical flow motion vector v at the first pixel position can be converted to 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0251] If the depth value of the first pixel position differs from the depth value of the surrounding pixels by more than or equal to the specified threshold 2, or the optical flow motion vector of the first pixel position differs from the optical flow motion vector of the surrounding pixels by more than or equal to the specified threshold 3, the optical flow motion vector v at the first pixel position can be 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0252] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S1010 may be performed.

[0253] S1010: The frame insertion module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0254] For the specific content, please refer to step S407 in the embodiment shown in FIG4 , which will not be described in detail here.

[0255] Therefore, an embodiment of the present application provides an image motion estimation method that uses the transformation matrix and depth information of two adjacent image frames, integrates optical flow motion estimation and 3D motion field motion estimation, and performs motion estimation on two adjacent image frames to determine the motion vector information between the two adjacent image frames. This can improve the accuracy of the motion estimation results.

[0256] In the above-mentioned embodiment of the present application, the motion estimation module can first determine whether the native motion vector information of the first application can be obtained from the graphics interface. If the native motion vector information of the first application can be obtained from the graphics interface, the method embodiment shown in the above-mentioned Figure 4 is executed; if the native motion vector information of the first application cannot be obtained from the graphics interface, the method embodiment shown in the above-mentioned Figure 9 is executed.

[0257] Optionally, the motion estimation module may first determine whether the native motion vector information of the first application and the keyword of the rendering object can be obtained from the graphics interface. If the native motion vector information of the first application and the keyword of the rendering object can be obtained from the graphics interface, the method embodiment shown in FIG4 is executed. If the keyword of the rendering object can be obtained, but the native motion vector information of the first application cannot be obtained from the graphics interface, the method embodiment shown in FIG9 is executed. If neither the keyword of the rendering object nor the native motion vector information of the first application can be obtained from the graphics interface, the method embodiment shown in FIG10 is executed.

[0258] In an embodiment of the present application, the motion vector information between two adjacent image frames determined by the above-mentioned image motion estimation method can be used in addition to image frame interpolation in game applications, and can also be used in other scenarios, which are not limited in the embodiments of the present application. For example, image frame interpolation in XR applications, and moving object prediction in autonomous driving. Among them, in XR applications or autonomous driving scenarios, two adjacent image frames can be obtained by a camera, and the depth information of the two adjacent image frames can be collected by a depth sensor, and then the motion field information and optical flow information in the two adjacent image frames can be determined. For another example, the motion vector information between two adjacent image frames determined by the above-mentioned image motion estimation method can also be used to perform motion estimation in game applications that do not have their own motion blur function, thereby realizing the motion blur function.

[0259] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0260] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0261] The present application also provides a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0262] The term "user interface (UI)" in the specification and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0263] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0264] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0265] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0266] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for image motion estimation, characterized in that: Applied to an electronic device, the electronic device having a first application installed therein, the method comprising: Obtaining a rendering instruction issued by the first application, where the rendering instruction is used to instruct a graphics processor (GPU) to render image frames of the first application, where the image frames of the first application include a first image frame and a second image frame; Identifying, from the rendering instruction, a pixel position of a first object in the first image frame and a pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; Identify, from the rendering instruction, a pixel position of the first object in the second image frame and a pixel position of the second object in the second image frame; Determine, by optical flow motion estimation method, a motion vector of a pixel position where the first object is located between the first image frame and the second image frame; Native motion vector information of the first application is obtained, and a motion vector of a pixel position where the second object is located between the first image frame and the second image frame is determined from the native motion vector information.

2. The method according to claim 1, characterized in that The electronic device includes a motion estimation module; The obtaining of the rendering instruction issued by the first application specifically includes: The motion estimation module obtains the rendering instruction issued by the first application from the graphics interface; The acquiring native motion vector information of the first application, and determining the motion vector of the pixel position where the second object is located between the first image frame and the second image frame from the native motion vector information specifically includes: The motion estimation module obtains native motion vector information of the first application from the graphic interface, and determines a motion vector of a pixel position where the second object is located between the first image frame and the second image frame from the native motion vector information.

3. The method according to claim 2, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines optical flow motion vector information between the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; The motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

4. The method according to claim 2, characterized in that: The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines an optical flow motion vector of a pixel position of the first object between the first image frame and the second image frame based on a pixel position of the first object in the first image frame, a pixel position of the second object in the second image frame, the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame to the motion estimation module; The motion estimation module uses the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame as the motion vector of the pixel position where the first object is located between the first image frame and the second image frame.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: The motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; The motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; The optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

6. The method according to any one of claims 1 to 5, characterized in that The native motion vector information is motion field motion vector information between the first image frame and the second image frame determined by the first application through a 3D motion field estimation method.

7. The method according to any one of claims 1 to 6, characterized in that The electronic device further includes a frame insertion module; and the method further includes: The motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; The interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

8. The method according to any one of claims 1 to 7, characterized in that Before identifying the pixel position of the first object in the first image frame and the pixel position of the second object in the first image frame from the rendering instruction, the method further includes: The motion estimation module determines that the native motion vector information can be obtained from the graphics interface.

9. The method according to any one of claims 1 to 7, characterized in that: The first object is a semi-transparent object, and the second object is an opaque object.

10. A method for image motion estimation, characterized in that: Applied to an electronic device, the electronic device is installed with a first application; the method comprises: Obtaining a rendering instruction issued by the first application, where the rendering instruction is used to instruct the GPU to render image frames of the first application, where the image frames of the first application include a first image frame and a second image frame; Identifying, from the rendering instruction, a pixel position of a first object in the first image frame and a pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; Identify, from the rendering instruction, a pixel position of the first object in the second image frame and a pixel position of the second object in the second image frame; Determine, by optical flow motion estimation method, a motion vector of a pixel position where the first object is located between the first image frame and the second image frame; Acquire a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame; Based on the first transformation matrix, the first depth information, the second transformation matrix and the second depth information, a motion vector of a pixel position where the second object is located between the first image frame and the second image frame is determined by a 3D motion field motion estimation method.

11. The method according to claim 10, characterized in that The electronic device includes a motion estimation module; The obtaining of the rendering instruction issued by the first application specifically includes: The motion estimation module obtains the rendering instruction issued by the first application from the graphics interface; The acquiring a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame specifically includes: The motion estimation module obtains a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame from the graphic interface.

12. The method according to claim 11, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines optical flow motion vector information between the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; The motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

13. The method according to claim 11, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines an optical flow motion vector of a pixel position of the first object between the first image frame and the second image frame based on a pixel position of the first object in the first image frame, a pixel position of the first object in the second image frame, the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame to the motion estimation module; The motion estimation module uses the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame as the motion vector of the pixel position where the first object is located between the first image frame and the second image frame.

14. The method according to claim 12 or 13, characterized in that The method further comprises: The motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; The motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; The optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

15. The method according to any one of claims 10 to 14, characterized in that The electronic device further includes a frame insertion module; and the method further includes: The motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; The interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

16. The method according to any one of claims 10 to 15, characterized in that The first object is a dynamic object, and the second object is a static object.

17. An electronic device, characterized in that: It comprises one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer programs, and when the one or more processors execute the computer programs, the electronic device executes the method as described in any one of claims 1-9 or the method as described in any one of claims 10-16.

18. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-9 or the method as described in any one of claims 10-16.

19. A computer-readable storage medium, characterized in that: The method comprises a computer program, and when the computer program is executed on a processor on an electronic device, the electronic device executes the method according to any one of claims 1 to 9 or the method according to any one of claims 10 to 16.

Images